-
Notifications
You must be signed in to change notification settings - Fork 2k
[ENH] Cancellation safety for append_batch. #4959
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
In a commit back in March I took out the cancellation safety from append batch. This re-adds that so that the batch only waits for itself, not until all work is done.
|
Restore Cancellation Safety to append_batch Processing This PR reintroduces cancellation safety for batch appends in the WAL writer. Now, the batch processing only waits for its own completion instead of until all work is done, by spawning each append_batch using tokio tasks. Minimal wording updates were made to clarify a comment in batch_manager.rs. Key Changes: Affected Areas: This summary was automatically generated by @propel-code-bot |
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
Sicheng-Pan
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: previously append_batch may continue to do more work, but after the fix it will only wait for its own work to be done.
## Description of changes In a commit back in March I took out the cancellation safety from append batch. This re-adds that so that the batch only waits for itself, not until all work is done. ## Test plan CI ## Documentation Changes N/A
## Description of changes Included changes - **[ENH] Purge dirty log in background at the end of scheduled compaction (#4915)** - **[ENH] Move Log GC to operator (#4919)** - **[BUG] Do not leak tokio tasks in the log service. (#4936)** - **[BUG] Log GC offset should be one above minimum compaction offset (#4938)** - **[ENH] Make roll dirty log always converge to coalesce everything. (#4927)** - **[BUG] Coalesce when multiple collections return the same info to compact (#4946)** - **[BUG] Enrich from the manifest if a cursor doesn't exist. (#4947)** - **[ENH] If the dirty log fails with LogContentionDurable, do not fail the operation. (#4953)** - **[ENH] Warn, not error, if dirty log has no cursor. (#4952)** - **[ENH] Cancellation safety for append_batch. (#4959)** ## Test plan CI ## Documentation Changes N/A --------- Co-authored-by: Macronova <[email protected]>
## Description of changes In a commit back in March I took out the cancellation safety from append batch. This re-adds that so that the batch only waits for itself, not until all work is done. ## Test plan CI ## Documentation Changes N/A
Description of changes
In a commit back in March I took out the cancellation safety from append batch.
This re-adds that so that the batch only waits for itself, not until all work is done.
Test plan
CI
Documentation Changes
N/A