Skip to content

Conversation

@codetheweb
Copy link
Contributor

@codetheweb codetheweb commented Jul 16, 2025

Description of changes

Cumulatively, this operation is very slow in production. Hoping that increasing the batch size to the maximum will speed it up.

Test plan

How are these changes tested?

  • Tests pass locally with pytest for python, yarn test for js, cargo test for rust

@github-actions
Copy link

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

  • Can you think of any use case in which the code does not behave as intended? Have they been tested?
  • Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
  • If appropriate, are there adequate property based tests?
  • If appropriate, are there adequate unit tests?
  • Should any logging, debugging, tracing information be added or removed?
  • Are error messages user-friendly?
  • Have all documentation changes needed been made?
  • Have all non-obvious changes been commented?

System Compatibility

  • Are there any potential impacts on other parts of the system or backward compatibility?
  • Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

  • Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

@codetheweb codetheweb marked this pull request as ready for review July 16, 2025 20:21
@codetheweb codetheweb requested a review from Sicheng-Pan July 16, 2025 20:21
@propel-code-bot
Copy link
Contributor

This pull request increases the batch size for file deletions in the garbage collector from 100 to 1,000 per chunk, taking advantage of the S3 DeleteObjects API's maximum allowed batch size. The intent is to improve deletion performance in production environments.

This summary was automatically generated by @propel-code-bot

@codetheweb codetheweb enabled auto-merge (squash) July 16, 2025 20:44
@codetheweb codetheweb merged commit 640b1d0 into main Jul 16, 2025
114 of 116 checks passed
@blacksmith-sh blacksmith-sh bot deleted a comment from codetheweb Jul 16, 2025
@codetheweb codetheweb deleted the fix-increase-gc-delete-objects-chunk-size branch July 16, 2025 20:55
Inventrohyder pushed a commit to Inventrohyder/chroma that referenced this pull request Aug 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants