Bulk index findings and sequentially invoke auto-correlations

[goyamegh]

Issue #, if available:
#1333

Description of changes:
All the findings once generated, are indexed one-by-one today. This PR attempts to optimize this by making a bulk index request for all the generated findings. Once completed, each finding will then be published to kick-start auto-correlations subsequently.

CheckList:

• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[eirsep]

NIT: plz add TODO/FIXME and maintain pr in draft if it's not ready to merge

[eirsep]

what is the previous behaviour? why are we not just refactoring? is there a behaviour change?

[goyamegh]

This change is preserving the existing behavior. Left by mistake, removed the comment.

[eirsep]

min cant be 0

[eirsep]

nit:BULK

[eirsep]

what is this - monitorCtx.findingsIndexBatchSize??

Why are we mutating value of monitorCtx object?? that object should only be read from

[goyamegh]

The intent is to update this value in alerting, when it is updated via _cluster/settings.

[eirsep]

can we test the new setting?

[eirsep]

we aren't supposed to add settings consumer in the middle of monitor execution. plz do at node startup

[goyamegh]

Moved it to registerConsumers() in MonitorRunnerService.

[eirsep]

dont refresh for every batch. refresh findings index only once after all batches are bulk-ingested

[goyamegh]

Done.

[eirsep]

plz add a test where you make bulk finding size =2 and simulate creation of 10 findings

[eirsep]

fix ktlint
CIs are failing.

[opensearch-trigger-bot]

The backport to 2.9 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.9 2.9
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.9
# Create a new branch
git switch --create backport-1355-to-2.9
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.9
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.9

Then, create a pull request where the base branch is 2.9 and the compare/head branch is backport-1355-to-2.9.

[opensearch-trigger-bot]

The backport to 2.8 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.8 2.8
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.8
# Create a new branch
git switch --create backport-1355-to-2.8
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.8
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.8

Then, create a pull request where the base branch is 2.8 and the compare/head branch is backport-1355-to-2.8.

[opensearch-trigger-bot]

The backport to 2.7 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.7 2.7
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.7
# Create a new branch
git switch --create backport-1355-to-2.7
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.7
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.7

Then, create a pull request where the base branch is 2.7 and the compare/head branch is backport-1355-to-2.7.

[opensearch-trigger-bot]

The backport to 2.6 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.6 2.6
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.6
# Create a new branch
git switch --create backport-1355-to-2.6
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.6
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.6

Then, create a pull request where the base branch is 2.6 and the compare/head branch is backport-1355-to-2.6.

[opensearch-trigger-bot]

The backport to 2.5 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.5 2.5
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.5
# Create a new branch
git switch --create backport-1355-to-2.5
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.5
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.5

Then, create a pull request where the base branch is 2.5 and the compare/head branch is backport-1355-to-2.5.

[opensearch-trigger-bot]

The backport to 2.11 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/alerting/backport-2.11 2.11
# Navigate to the new working tree
pushd ../.worktrees/alerting/backport-2.11
# Create a new branch
git switch --create backport-1355-to-2.11
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b56196557b539b2f6069dc407f301cd9c15771ea
# Push it to GitHub
git push --set-upstream origin backport-1355-to-2.11
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/alerting/backport-2.11

Then, create a pull request where the base branch is 2.11 and the compare/head branch is backport-1355-to-2.11.