Upload global cluster state to remote store

[dhwanilpatel]

Description

Changes for uploading global metadata to remote store.

We will store all global metadata in single file in remote store. For incremental update we will check if global metadata has changed or not, if not then we will skip uploading it and if it has changed we will upload full global metadata to remote store.

Directory structure will look like this:

base folder/
    |
    |--> index/
    |     | --> index_UUID/
    |              | --> metadata__<inverted_index_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |              | --> metadata__<inverted_index_metadata_version>__<inverted_codec_version>__<timestamp>.dat  
    |
    |--> global-metadata/
    |       | --> metadata__<inverted_metadata_version>__<inverted_codec_version>__<timestamp>.dat
    |
    |
    |--> manifest/
    |       | --> manifest__<inverted_term>__<inverted_version>__<inverted_codec_version>__<timestamp>
    |       | --> manifest__<inverted_term>__<inverted_version>__<inverted_codec_version>__<timestamp>

Related Issues

#10526

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• GitHub issue/PR created in OpenSearch documentation repo for the required public documentation changes (#[Issue/PR number])

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27124/
• CommitID: 3c6a3b0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 1aa7bcf

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

[Bukhtawar]

Thanks Dhwanil. Can you show the complete directory structure including index metadata to get a sense of the blob structure

[dhwanilpatel]

Thanks Dhwanil. Can you show the complete directory structure including index metadata to get a sense of the blob structure

This will be sample directory structure:

base folder/
    |
    |--> index/
    |     | --> index_UUID/
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat  
    |
    |--> global-metadata/
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |
    |
    |--> manifest/
    |       | --> manifest__<term>__<version>__<timestamp>
    |       | --> manifest__<term>__<version>__<timestamp>

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27275/
• CommitID: d0d1ccf
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[shwetathareja]

Thanks Dhwanil. Can you show the complete directory structure including index metadata to get a sense of the blob structure

This will be sample directory structure:

base folder/
    |
    |--> index/
    |     | --> index_UUID/
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat  
    |
    |--> global-metadata/
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |
    |
    |--> manifest/
    |       | --> manifest__<term>__<version>__<timestamp>
    |       | --> manifest__<term>__<version>__<timestamp>

@dhwanilpatel what is the codec version here? The global metadata file should contain Metadata version as well.

[Bukhtawar]

Thanks Dhwanil. Can you show the complete directory structure including index metadata to get a sense of the blob structure

This will be sample directory structure:

base folder/
    |
    |--> index/
    |     | --> index_UUID/
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat
    |              | --> metadata__<index_metadata_version>__<timestamp>.dat  
    |
    |--> global-metadata/
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |       | --> meta-global-metadata__<codec_version>__<timestamp>.dat
    |
    |
    |--> manifest/
    |       | --> manifest__<term>__<version>__<timestamp>
    |       | --> manifest__<term>__<version>__<timestamp>

@dhwanilpatel what is the codec version here? The global metadata file should contain Metadata version as well.

I guess the codec version here is the format of the meta-global-metadata file for future extensibility. The Metadata version will be serialised into the XContent format and should be available when reading?

[shwetathareja]

The Metadata version will be serialised into the XContent format and should be available when reading?

The Metadata file path should contain the version as well, timestamp is just for preventing overwrites in case of multiple uploads. Also why do you need a separate codec version when cluster version is already written in manifest file.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27433/
• CommitID: 55839b4
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[Bukhtawar]

The Metadata version will be serialised into the XContent format and should be available when reading?

The Metadata file path should contain the version as well, timestamp is just for preventing overwrites in case of multiple uploads. Also why do you need a separate codec version when cluster version is already written in manifest file.

We discussed this offline and think, would be better to revisit versioning, essentially each file needs to have the file format versioned and the file format version should be surfaced to the file name to ensure we know how to parse the file if the underlying file format changes in future. Essentially having two different versions

1. File format version — Applicable to manifest and other files where lets say format of the manifest file changes. To handle BWC from a pure file parsing logic
2. Index/Engine/Cluster metadata compatibility version — To handle BWC at the engine level or maintaining any internal state consistency specific to the internal working of OpenSearch

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/27448/
• CommitID: c2cf656
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[shwetathareja]

@Bukhtawar @shwetathareja we don't have garbage collection for dangling files of index metadata and global metadata where corresponding manifest upload fails. it will only be eligible for GC when the cluster uuid changes after quorum loss recovery. Should we add a task for this?

@linuxpi yes lets create an issue

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28207/
• CommitID: 03167ad
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28209/
• CommitID: 5d09048
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28213/
• CommitID: 00474a9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[shwetathareja]

finishing the tests, publishing other comments.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/28223/
• CommitID: 7f92d59

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}

• URL: https://build.ci.opensearch.org/job/gradle-check/28232/
• CommitID: 1aa7bcf
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.