Update last seen cluster state in commit phase

[soosinha]

Description

Coordination metadata contains accepted voting configuration and committed voting configuration.
When the voting configuration changes, the changed accepted voting configuration is sent in publish phase. The follower nodes apply this publish phase cluster state on the lastSeen cluster state in PublicationTransportHandler. In the commit phase, each node set the committed voting configuration using the accepted voting configuration by themselves.
When the next cluster state is published, it does not contain any diff in the voting configuration. This next cluster state update uses lastSeen again to apply the diff due to which the committed voting configuration is still set to an older value.
With local cluster state publication, this issue does not occur as the entire coordination metadata is sent in every diff.
With remote cluster state publication, we can mitigate this in the same way by always sending coordination metadata in the diff. But this has the overhead of downloading the coordination metadata everytime on each data node.
So the alternative approach is to update the lastSeen cluster state in the commite phase

Related Issues

NA

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 3a1e37f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Bukhtawar]

The only concern I see is the additional download overhead(remote store calls) on the follower nodes

[soosinha]

The only concern I see is the additional download overhead(remote store calls) on the follower nodes

I think the long term fix should be to update the lastSeen cluster state in commit phase with the committed config.

[github-actions]

❕ Gradle check result for 8ffcaed: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.03%. Comparing base (9ddee61) to head (86d2ee2).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #16215   +/-   ##
=========================================
  Coverage     72.03%   72.03%           
- Complexity    64782    64818   +36     
=========================================
  Files          5307     5307           
  Lines        302545   302548    +3     
  Branches      43703    43703           
=========================================
+ Hits         217925   217940   +15     
- Misses        66712    66745   +33     
+ Partials      17908    17863   -45     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dbwiddis]

LGTM but needs a CHANGELOG entry

[github-actions]

✅ Gradle check result for 86d2ee2: SUCCESS

[soosinha]

LGTM but needs a CHANGELOG entry

Added CHANGELOG

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-16215-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a53e0c63aa9c6a85fed30e7bdce8b533aa471060
# Push it to GitHub
git push --set-upstream origin backport/backport-16215-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-16215-to-2.x.