Add looped mode to bulk operation#1830
Merged
gbanasiak merged 1 commit intoelastic:masterfrom Feb 21, 2024
Merged
Conversation
Contributor
Author
|
Re-running IT test failure of which seems unrelated. |
inqueue
approved these changes
Feb 20, 2024
| # start from the beginning in looped mode, otherwise stop the run | ||
| if self.looped: | ||
| self.current_bulk = 0 | ||
| self._init_internal_params() |
Member
There was a problem hiding this comment.
With looped mode, we will have the possibility to add an override for bulk and batch sizes in steps if it is something we can use in the future.
Edit: I see bulk size would be relatively easy to adjust in the task definition itself, and not necessarily here directly.
Contributor
Author
There was a problem hiding this comment.
The idea is looped operation should be limited at the task level, and you can have as many tasks with different batch sizes, etc. as you want. Looped mode basically means "keep on generating bulk requests until some external criteria stop the task".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
loopedmode to bulk operation. That is useful in benchmarks where we want Rally to ingest for a specific amount of time even if corpus is not big enough to sustain indexing at a specific rate for this time.This mode is meant to be used together with
time-periodoriterationsat the task level. Otherwise Rally will never complete the task.Example use: