Skip to content

Add looped mode to bulk operation#1830

Merged
gbanasiak merged 1 commit intoelastic:masterfrom
gbanasiak:bulk-looped-mode
Feb 21, 2024
Merged

Add looped mode to bulk operation#1830
gbanasiak merged 1 commit intoelastic:masterfrom
gbanasiak:bulk-looped-mode

Conversation

@gbanasiak
Copy link
Copy Markdown
Contributor

@gbanasiak gbanasiak commented Feb 20, 2024

Adds looped mode to bulk operation. That is useful in benchmarks where we want Rally to ingest for a specific amount of time even if corpus is not big enough to sustain indexing at a specific rate for this time.

This mode is meant to be used together with time-period or iterations at the task level. Otherwise Rally will never complete the task.

Example use:

{
  "name": "some-bulk-task",
  "operation": {
    "operation-type": "bulk",
    "bulk-size": 5000,
    "looped": true
  },
  "time-period": 300,
  "target-throughput": 2
}

@gbanasiak gbanasiak added the enhancement Improves the status quo label Feb 20, 2024
@gbanasiak gbanasiak requested a review from a team February 20, 2024 11:03
@gbanasiak
Copy link
Copy Markdown
Contributor Author

Re-running IT test failure of which seems unrelated.

Copy link
Copy Markdown
Member

@inqueue inqueue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# start from the beginning in looped mode, otherwise stop the run
if self.looped:
self.current_bulk = 0
self._init_internal_params()
Copy link
Copy Markdown
Member

@inqueue inqueue Feb 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With looped mode, we will have the possibility to add an override for bulk and batch sizes in steps if it is something we can use in the future.

Edit: I see bulk size would be relatively easy to adjust in the task definition itself, and not necessarily here directly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea is looped operation should be limited at the task level, and you can have as many tasks with different batch sizes, etc. as you want. Looped mode basically means "keep on generating bulk requests until some external criteria stop the task".

@gbanasiak gbanasiak merged commit 43e56f6 into elastic:master Feb 21, 2024
@gbanasiak gbanasiak deleted the bulk-looped-mode branch February 21, 2024 12:09
@gareth-ellis gareth-ellis added this to the 2.11 milestone May 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Improves the status quo

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants