Skip to content

[Remote Store] Changes to perform repository registration during bootstrap via node attributes.#9105

Merged
gbbafna merged 34 commits intoopensearch-project:mainfrom
psychbot:remote-repo-registration
Sep 6, 2023
Merged

[Remote Store] Changes to perform repository registration during bootstrap via node attributes.#9105
gbbafna merged 34 commits intoopensearch-project:mainfrom
psychbot:remote-repo-registration

Conversation

@psychbot
Copy link
Copy Markdown
Member

@psychbot psychbot commented Aug 3, 2023

Description

Solves #8623
The changes are to perform repository registration via the values supplied in node attributes during node bootstap.
Reject or accept the join request if they have matchin node attributes.

Below are the format in which the node attributes will be supplied.

Config 1

node.attr.remote_store.segment.repository : remote-store-A
node.attr.remote_store.repository.remote-store-A.type : s3
node.attr.remote_store.repository.remote-store-A.settings : 
        bucket : abc
        base_path : xyz
node.attr.remote_store.translog.repository : remote-store-B
node.attr.remote_store.repository.remote-store-B.type : hdfs
node.attr.remote_store.repository.remote-store-B.settings : 
        uri : blah
        base_path : foo

Config 2

node.attr.remote_store.segment.repository : remote-store-A
node.attr.remote_store.repository.remote-store-A.type : s3
node.attr.remote_store.repository.remote-store-A.settings : 
        bucket : abc
        base_path : xyz
node.attr.remote_store.translog.repository : remote-store-A

Related

Resolves #8623

Pending Items

  • Error Handling
  • Repository Verification
  • Unit Tests
  • Integ Tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Aug 3, 2023

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 28m 9s

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 25m 27s

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@Bukhtawar
Copy link
Copy Markdown
Contributor

Bukhtawar commented Aug 12, 2023

How about simplifying the configurations to

      - cluster.system_repositories.segment: "remote-segment-repo"
      - cluster.system_repositories.translog: "remote-translog-repo"
      - node.attr.remote-segment-repo :
            - type : s3
               settings
                    - bucket : bucket_name
                       base_path : xyz/abc
      - node.attr.remote-translog-repo :
            - type : s3
               settings
                    - bucket : bucket_name
                       base_path : xyz/abc

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 27m 35s

@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Copy Markdown
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pls add more comments on the places where we are updating the cluster state. Specifically where a condition evaluates to true and we update the state. It would help understand under what circumstances it would update the state.

@psychbot psychbot force-pushed the remote-repo-registration branch from e89fbcf to bcf8ef3 Compare August 17, 2023 16:39
@github-actions
Copy link
Copy Markdown
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

Compatibility status:

Checks if related components are compatible with change 6a5b464

Incompatible components

Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@psychbot psychbot force-pushed the remote-repo-registration branch from bcf8ef3 to 5ada3cd Compare August 17, 2023 18:56
Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 6, 2023

Compatibility status:

Checks if related components are compatible with change 8c54a86

Incompatible components

Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/neural-search.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Sep 6, 2023

Gradle Check (Jenkins) Run Completed with:

@gbbafna gbbafna merged commit 9119b6d into opensearch-project:main Sep 6, 2023
@gbbafna
Copy link
Copy Markdown
Contributor

gbbafna commented Sep 6, 2023

The test org.opensearch.search.profile.query.QueryProfilerIT.testProfileMatchesRegular {p0={"search.concurrent_segment_search.enabled":"true"} p1=true} passed in the previous run and failed in this and is flaky .

@psychbot
Copy link
Copy Markdown
Member Author

psychbot commented Sep 6, 2023

A different test is flaky now - org.opensearch.search.profile.query.QueryProfilerIT.testProfileMatchesRegular {p0={"search.concurrent_segment_search.enabled":"true"} p1=true}
#9787

@gbbafna gbbafna added the backport 2.x Backport to 2.x branch label Sep 6, 2023
@opensearch-trigger-bot
Copy link
Copy Markdown
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9105-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 9119b6dc20ea11d95a399c68505f1d858b78e30e
# Push it to GitHub
git push --set-upstream origin backport/backport-9105-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9105-to-2.x.

@psychbot psychbot deleted the remote-repo-registration branch September 6, 2023 10:29
psychbot pushed a commit to psychbot/OpenSearch that referenced this pull request Sep 6, 2023
…on during bootstrap via node attributes. (opensearch-project#9105)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
(cherry picked from commit 9119b6d)
gbbafna pushed a commit that referenced this pull request Sep 6, 2023
…on during bootstrap via node attributes. (#9105) (#9809)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
(cherry picked from commit 9119b6d)
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…strap via node attributes. (opensearch-project#9105)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…strap via node attributes. (opensearch-project#9105)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…strap via node attributes. (opensearch-project#9105)

Signed-off-by: Dharmesh 💤 <sdharms@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
// use these joins to try and become the cluster-manager.
// Note that we don't have to do any validation of the amount of joining nodes - the commit
// during the cluster state publishing guarantees that we have enough
newState = becomeClusterManagerAndTrimConflictingNodes(currentState, joiningNodes);
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could update routingTable and metadata.indices["index-name"].primaryTerms for the primary shards in deleted nodes. Are we overwriting this metadata update in this PR at multiple places by using currentState.metadata() instead of newState.metadata()?

This breaks the assertion at https://github.com/opensearch-project/OpenSearch/blob/2.19/server/src/main/java/org/opensearch/index/shard/IndexShard.java#L846-L850 in the node containing the replica that is to be promoted (according to the routingTable update. But the metadata is stale)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport 2.x Backport to 2.x branch backport-failed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RFC : Repository Registration for Remote Backed Storage

7 participants