[ENH]: Add ability to set different block sizes for different blockfiles

[sanketkedia]

Description of changes

Summarize the changes made by this PR.

• Improvements & Bug fixes
  • We persist the max_block_size_bytes in the arrow metadata now and also add a new blockfile version for handling backward compatibility with old data
  • This is need so that spann posting list can have a block size different from other blockfiles
  • This PR also sets the spann PL block size to 5MiB
• New functionality
  • ...

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

None

[sanketkedia]

• [ENH]: Add ability to set different block sizes for different blockfiles #4948 👈 (View in Graphite)
• [ENH]: granular lock for posting list #4937
• [ENH]: More concurrent blockfilewriter #4889
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[propel-code-bot]

Enable Per-Blockfile Custom Block Sizes and Metadata Migration

This PR introduces support for specifying different max block sizes for individual blockfiles. The blockfile versioning and Arrow metadata schema are updated to persist block size information per file, with corresponding migration logic to maintain backward compatibility. The primary motivator is to allow the Spann posting list blockfiles to use a distinct block size (defaulted to 5MiB) while leaving the default 8MiB setting for other blockfile types. Extensive tests and migration scenarios are included to ensure correctness and compatibility across prior versions.

Key Changes:
• Blockfile writer and APIs updated to allow specifying max_block_size_bytes per blockfile via options object
• Arrow blockfile root metadata now persists max_block_size_bytes in each file, with new version V1_2 introduced
• BlockManager now exposes default_max_block_size_bytes, passed down to blockfile readers/writers as fallback
• Arrow (ordered/unordered) blockfile writers/readers and provider now use per-file block size instead of global default
• Spann Posting List creation now specifies a custom 5MiB block size (overridable via config)
• Blockfile migration system enhanced to handle transitions between blockfile versions, including V1_1→V1_2 upgrade and correct block size hydration
• Extensive migration and backward compatibility tests ensuring various edge cases are handled (legacy, partially upgraded, fully upgraded files)
• Tilt and config files updated to allow configurable pl_block_size for Spann in deployment
• API and config plumbing for passing custom block size through segment/provider layers

Affected Areas:
• rust/blockstore/src/arrow/blockfile.rs
• rust/blockstore/src/arrow/ordered_blockfile_writer.rs
• rust/blockstore/src/arrow/provider.rs
• rust/blockstore/src/arrow/root.rs
• rust/blockstore/src/arrow/migrations.rs
• rust/blockstore/src/types/writer_options.rs
• rust/index/src/config.rs
• rust/index/src/spann/types.rs
• rust/segment/src/spann_provider.rs
• rust/segment/src/distributed_spann.rs
• rust/segment/src/test.rs
• rust/worker/tilt_config.yaml

This summary was automatically generated by @propel-code-bot

[HammadB]

Should add a justification for why this

[codetheweb]

should update this method name