[BUG] Cannot use remote store-based segment replication without enabling remote cluster state

[msfroh]

Describe the bug

The instructions at https://docs.opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/index/ to enable remote-backed storage don't work anymore.

In #9811, there was a change made to say that all three of remote_store.segment.repository, remote_store.translog.repository, and remote_store.state.repository must be set. A few months later, #11858 said, "Wait a second -- what if I don't want to enable all three?" and so it allowed you to specify remote_store.state.repository without the other two.

But what if I want to store index data in the remote store without storing cluster state in the remote store? This looks like a regression from whenever the documentation was written, since the example in the docs doesn't work anymore.

Related component

Storage:Remote

To Reproduce

1. Follow instructions from https://docs.opensearch.org/docs/latest/tuning-your-cluster/availability-and-recovery/remote-store/index/
2. Try to start a node
3. Receive the following error:

 java.lang.IllegalStateException: joining node [{runTask-0}{w6uWEy1gQ_aaOPPDa4xe-g}{2XCOT6rOSjKlu4iHbuqe_g}{127.0.0.1}{127.0.0.1:9300}{dimr}{testattr=test, shard_indexing_pressure_enabled=true}] doesn't have the node attribute [remote_store.state.repository]
»       at org.opensearch.node.remotestore.RemoteStoreNodeAttribute.validateAttributeNonNull(RemoteStoreNodeAttribute.java:103)
»       at org.opensearch.node.remotestore.RemoteStoreNodeAttribute.getValidatedRepositoryNames(RemoteStoreNodeAttribute.java:199)
»       at org.opensearch.node.remotestore.RemoteStoreNodeAttribute.buildRepositoriesMetadata(RemoteStoreNodeAttribute.java:174)
»       at org.opensearch.node.remotestore.RemoteStoreNodeAttribute.<init>(RemoteStoreNodeAttribute.java:88)
»       at org.opensearch.node.remotestore.RemoteStoreNodeService.createAndVerifyRepositories(RemoteStoreNodeService.java:139)
»       at org.opensearch.node.Node$LocalNodeFactory.apply(Node.java:2221)
»       at org.opensearch.node.Node$LocalNodeFactory.apply(Node.java:2200)
»       at org.opensearch.transport.TransportService.doStart(TransportService.java:313)
»       at org.opensearch.common.lifecycle.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:77)
»       at org.opensearch.node.Node.start(Node.java:1741)
»       at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:346)
»       at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:420)
»       at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
»       at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
»       at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
»       at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
»       at org.opensearch.cli.Command.main(Command.java:101)
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
»       at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)

Expected behavior

I should be able to enable remote store for shard data without enabling it for cluster state.

(Actually, I'd really like to only enable remote store for segments since I'm using search-only replicas and pull-based ingestion, so I don't care about persisting the translog. It looks like I lost that ability thanks to #8719.)

Additional Details

N/A

[msfroh]

For the record, I was able to tweak the validation to check remote cluster state separately from remote storage and everything worked, so it doesn't look like a massive change.

[msfroh]

Regarding my comment about the remote translog, I think I could add a node level setting to explicitly say that I don't want a remote translog. I'm happy with the default behavior saying that translog and segments go hand in hand, but as an expert, I want to decouple them.

[shwetathareja]

@msfroh The reason remote store and remote state was mandated to ensure durability guarantee (with no data loss) as primary term is stored in IndexMetadata. So its not a bug. But yes, if any user intentionally wants to turn off remote state and just need remote store, it should be reasonable to support it.

[msfroh]

Thanks, @shwetathareja!

I've opened a PR that adds a new option cluster.remote_store.segments_only to run without remote cluster state or remote translog. This addresses the needs for my project, but I haven't made it very flexible. That said, I think the only potentially valid case that's not covered is remote segments + translog, but no remote cluster state. (For example, I don't believe it's ever valid to enable remote translog without remote segments.)

[krisfreedain]

Catch All Triage - 1 2