[BUG] Searchable snapshot restore - null_pointer_exception

[xodat2]

Describe the bug

Hello,
Recently restoring snapshot as searchable is not working on our cluster.
I've tried to reproduce it on my local cluster, but it worked there...
When I change storage_type from remote_snapshot to local, snapshot is being restored without issue.
Can you please help with this ?

POST /_snapshot/azure_data_repository/retail-legacy-ppe-000337-2024.03.27-06:44:53.545/_restore { "indices": "*", "storage_type": "remote_snapshot", "ignore_unavailable": true, "include_global_state": false, "rename_pattern": "(.+)", "rename_replacement": "observability-searchable-sna2p-test2-$1", "include_aliases": false }

Response:

{ "error": { "root_cause": [ { "type": "null_pointer_exception", "reason": "Cannot invoke \"org.opensearch.index.IndexService.getIndexSettings()\" because the return value of \"org.opensearch.indices.IndicesService.indexService(org.opensearch.core.index.Index)\" is null" } ], "type": "null_pointer_exception", "reason": "Cannot invoke \"org.opensearch.index.IndexService.getIndexSettings()\" because the return value of \"org.opensearch.indices.IndicesService.indexService(org.opensearch.core.index.Index)\" is null" }, "status": 500 }

Logs:
org.opensearch.transport.RemoteTransportException: [observability-opensearch-master-1][***][cluster:admin/snapshot/restore] Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.index.IndexService.getIndexSettings()" because the return value of "org.opensearch.indices.IndicesService.indexService(org.opensearch.core.index.Index)" is null at org.opensearch.snapshots.RestoreService$1.lambda$validateSearchableSnapshotRestorable$5(RestoreService.java:809) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.routing.RoutingTable.allShardsSatisfyingPredicate(RoutingTable.java:325) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.routing.RoutingTable.allShardsSatisfyingPredicate(RoutingTable.java:307) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.snapshots.RestoreService$1.validateSearchableSnapshotRestorable(RestoreService.java:813) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.snapshots.RestoreService$1.execute(RestoreService.java:591) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:67) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:882) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:434) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:301) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:212) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:854) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:283) ~[opensearch-2.12.0.jar:2.12.0] at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:246) ~[opensearch-2.12.0.jar:2.12.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1583) [?:?]

Related component

Storage:Snapshots

To Reproduce

Restore snapshot as remove storage.

Expected behavior

Snapshot restored as remote snap

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: k8s
• Version 2.12.0

Additional context
Add any other context about the problem here.

[peternied]

Thanks for writing up this bug, we should fix all exceptions like this

[xodat2]

Thanks for writing up this bug, we should fix all exceptions like this

Hello,
Do you have any clue how can I debug this issue? are there debug/trace logs available for snapshots? I've tried to change log level for restore service logger, but it didn't throw any debug or trace logs.

[xodat2]

Hello,
I've discovered that when I stop our "search" nodes, it is possible to "restore" searchable snapshot and shards will be assigned after starting those nodes.

[ahmadbbi]

We are facing the same issue. Any solution or workaround other than stopping and starting search nodes?

[xodat2]

We are facing the same issue. Any solution or workaround other than stopping and starting search nodes?

Hello, were you able to reproduce this issue on different cluster?
We didn't find any solution, but it's hard to debug it on production cluster.

[ahmadbbi]

Hello, were you able to reproduce this issue on different cluster? We didn't find any solution, but it's hard to debug it on production cluster.

Yes, We use the opensearch operator to provision a cluster on K8s. I have provisioned multiple v2.12 clusters with ‘search’ nodes and all had the same issue. Even with v2.13 the issue exists. Wondering how are you handling that on your prod cluster?