[Data] Handle prefetches buffering in iter_batches#58657
Merged
Conversation
srinathk10
force-pushed
the
srinathk10/format_batches
branch
from
d3bfaf0 to
35dba95
Compare
srinathk10
changed the title
Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com>
srinathk10
force-pushed
the
srinathk10/format_batches
branch
from
71ef8c7 to
62ada1d
Compare
Contributor
Author
Microbenchmark for prefetches in iter_batches
Before After Benchmark script |
xinyuangui2
reviewed
Nov 17, 2025
xinyuangui2
reviewed
Nov 17, 2025
Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com>
srinathk10
changed the title
raulchen
reviewed
Nov 19, 2025
Co-authored-by: Hao Chen <chenh1024@gmail.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
raulchen
approved these changes
Nov 20, 2025
Contributor
Author
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request addresses an issue with buffering in iter_batches to better align with prefetching, aiming to provide more predictable latency. The changes introduce buffering in _iter_batches and _format_in_threadpool by setting the queue depth in make_async_gen based on the prefetch count. Additionally, the number of workers in the formatting threadpool is now capped to a default of 4 to optimize for latency.
The implementation looks solid and correctly applies the described fixes. I've found a minor but important typo in a newly introduced constant and its corresponding environment variable, which should be corrected before merging. My review includes suggestions to fix this.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com>
400Ping
pushed a commit
to 400Ping/ray
that referenced
this pull request
Nov 21, 2025
> Thank you for contributing to Ray! 🚀 > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. >⚠️ Remove these instructions before submitting your PR. > 💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. ### [Data] Handle prefetches buffering in iter_batches **Requirements** - Consumer of `iter_batches` requires predictable latency on batches, though overall Data Pipeline is optimized for maximizing throughput and utilization. - Essentially prefetching is used by the Consumer to impedance match Latency requirement with the Throughput optimized datapipeline. **Issue** - Queuing/buffering was not set up in `_iter_batches` to match prefetching. - Multiple Workers were set up for `_format_in_threadpool` as f(prefetch_count) which adds to latency variance on `_iter_batches`. **Fix** - In `_iter_batches`, set up queue depth for buffering in `make_async_gen` to honor prefetching. - In `_format_in_threadpool`, restrict to maximum of 4 workers by default, so as to optimize for `iter_batches` latency. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com> Co-authored-by: Hao Chen <chenh1024@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
ykdojo
pushed a commit
to ykdojo/ray
that referenced
this pull request
Nov 27, 2025
> Thank you for contributing to Ray! 🚀 > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. >⚠️ Remove these instructions before submitting your PR. > 💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. ### [Data] Handle prefetches buffering in iter_batches **Requirements** - Consumer of `iter_batches` requires predictable latency on batches, though overall Data Pipeline is optimized for maximizing throughput and utilization. - Essentially prefetching is used by the Consumer to impedance match Latency requirement with the Throughput optimized datapipeline. **Issue** - Queuing/buffering was not set up in `_iter_batches` to match prefetching. - Multiple Workers were set up for `_format_in_threadpool` as f(prefetch_count) which adds to latency variance on `_iter_batches`. **Fix** - In `_iter_batches`, set up queue depth for buffering in `make_async_gen` to honor prefetching. - In `_format_in_threadpool`, restrict to maximum of 4 workers by default, so as to optimize for `iter_batches` latency. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com> Co-authored-by: Hao Chen <chenh1024@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>
SheldonTsen
pushed a commit
to SheldonTsen/ray
that referenced
this pull request
Dec 1, 2025
> Thank you for contributing to Ray! 🚀 > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. >⚠️ Remove these instructions before submitting your PR. > 💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. ### [Data] Handle prefetches buffering in iter_batches **Requirements** - Consumer of `iter_batches` requires predictable latency on batches, though overall Data Pipeline is optimized for maximizing throughput and utilization. - Essentially prefetching is used by the Consumer to impedance match Latency requirement with the Throughput optimized datapipeline. **Issue** - Queuing/buffering was not set up in `_iter_batches` to match prefetching. - Multiple Workers were set up for `_format_in_threadpool` as f(prefetch_count) which adds to latency variance on `_iter_batches`. **Fix** - In `_iter_batches`, set up queue depth for buffering in `make_async_gen` to honor prefetching. - In `_format_in_threadpool`, restrict to maximum of 4 workers by default, so as to optimize for `iter_batches` latency. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com> Co-authored-by: Hao Chen <chenh1024@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
peterxcli
pushed a commit
to peterxcli/ray
that referenced
this pull request
Feb 25, 2026
> Thank you for contributing to Ray! 🚀 > Please review the [Ray Contribution Guide](https://docs.ray.io/en/master/ray-contribute/getting-involved.html) before opening a pull request. >⚠️ Remove these instructions before submitting your PR. > 💡 Tip: Mark as draft if you want early feedback, or ready for review when it's complete. ## Description > Briefly describe what this PR accomplishes and why it's needed. ### [Data] Handle prefetches buffering in iter_batches **Requirements** - Consumer of `iter_batches` requires predictable latency on batches, though overall Data Pipeline is optimized for maximizing throughput and utilization. - Essentially prefetching is used by the Consumer to impedance match Latency requirement with the Throughput optimized datapipeline. **Issue** - Queuing/buffering was not set up in `_iter_batches` to match prefetching. - Multiple Workers were set up for `_format_in_threadpool` as f(prefetch_count) which adds to latency variance on `_iter_batches`. **Fix** - In `_iter_batches`, set up queue depth for buffering in `make_async_gen` to honor prefetching. - In `_format_in_threadpool`, restrict to maximum of 4 workers by default, so as to optimize for `iter_batches` latency. ## Related issues > Link related issues: "Fixes ray-project#1234", "Closes ray-project#1234", or "Related to ray-project#1234". ## Additional information > Optional: Add implementation details, API changes, usage examples, screenshots, etc. --------- Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com> Signed-off-by: Srinath Krishnamachari <68668616+srinathk10@users.noreply.github.com> Co-authored-by: Hao Chen <chenh1024@gmail.com> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Signed-off-by: peterxcli <peterxcli@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
[Data] Handle prefetches buffering in iter_batches
Requirements
iter_batchesrequires predictable latency on batches, though overall Data Pipeline is optimized for maximizing throughput and utilization.Issue
_iter_batchesto match prefetching._format_in_threadpoolas f(prefetch_count) which adds to latency variance on_iter_batches.Fix
_iter_batches, set up queue depth for buffering inmake_async_gento honor prefetching._format_in_threadpool, restrict to maximum of 4 workers by default, so as to optimize foriter_batcheslatency.Related issues
Additional information