[ENH]: limit number of concurrent get_all_block_ids() when using buffer_unordered()

[codetheweb]

Description of changes

One of our GC pods is currently OOMing and it seems like the majority of memory used is just to track all the futures for get_all_block_ids():

This should not be an issue for the vast majority of collections, but there are currently 0.02% of collections with > 1k active versions and a handful with > 20k active versions.

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs section?

n/a

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

This PR limits the number of concurrent futures launched by buffer_unordered during execution of get_all_block_ids in the garbage collector's list_files_at_version operator. The concurrency cap is set to 100, replacing the previous unbounded use of sparse_index_ids.len(), to reduce memory usage and prevent out-of-memory (OOM) conditions for rare large collections.

This summary was automatically generated by @propel-code-bot