[BUG] Can't restore an index with sorting defined

[gr0]

Describe the bug

When an index with index sorting is being restored an error happens preventing it from being restored. It seems to be only happening for indices with sorting defined, the indices without it doesn't seem to be affected. In addition trying to restore with the index.sort.field and index.sort.order added to ignore_index_settings doesn't change the situation.

OpenSearch accepts the restore command and returns:

{"accepted":true}

But in the logs we can see:

[2025-11-21T15:42:59,154][WARN ][o.o.c.r.a.AllocationService] [xarea.local] failing shard [failed shard, shard [test][0], node[I7HYk4sKTm2Hm7i__wxf2g], [P], recovery_source[snapshot recovery [rxwlTnTXSo-1UHY43a9M9g] from test-fs-repository:1/1jJfc8QJQxSFRMgnopbSfg], s[INITIALIZING], a[id=r2OwMITXR1OK8RHlpXm7Tw], unassigned_info[[reason=ALLOCATION_FAILED], at[2025-11-21T14:42:59.102Z], failed_attempts[4], failed_nodes[[I7HYk4sKTm2Hm7i__wxf2g]], delayed=false, details[failed shard on node [I7HYk4sKTm2Hm7i__wxf2g]: failed recovery, failure RecoveryFailedException[[test][0]: Recovery failed on {xarea.local}{I7HYk4sKTm2Hm7i__wxf2g}{1MF4EKFlToqU9LouSSAKUQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [1/1jJfc8QJQxSFRMgnopbSfg]]; nested: IllegalArgumentException[can't add field [__nested_parent] as parent document field; this IndexWriter has no parent document field configured]; ], allocation_status[fetching_shard_data]], expected_shard_size[4304], message [failed recovery], failure [RecoveryFailedException[[test][0]: Recovery failed on {xarea.local}{I7HYk4sKTm2Hm7i__wxf2g}{1MF4EKFlToqU9LouSSAKUQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}]; nested: IndexShardRecoveryException[failed recovery]; nested: IndexShardRestoreFailedException[restore failed]; nested: IndexShardRestoreFailedException[failed to restore snapshot [1/1jJfc8QJQxSFRMgnopbSfg]]; nested: IllegalArgumentException[can't add field [__nested_parent] as parent document field; this IndexWriter has no parent document field configured]; ], markAsStale [true]]
org.opensearch.indices.recovery.RecoveryFailedException: [test][0]: Recovery failed on {xarea.local}{I7HYk4sKTm2Hm7i__wxf2g}{1MF4EKFlToqU9LouSSAKUQ}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
	at org.opensearch.index.shard.IndexShard.lambda$executeRecovery$1(IndexShard.java:4132) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.index.shard.StoreRecovery.lambda$recoveryListener$1(StoreRecovery.java:639) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.index.shard.StoreRecovery.lambda$restore$1(StoreRecovery.java:918) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$0(BlobStoreRepository.java:4172) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$2.onFailure(ActionListener.java:113) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84) ~[opensearch-core-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.BlobStoreRepository$10.restoreFiles(BlobStoreRepository.java:4190) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.FileRestoreContext.restore(FileRestoreContext.java:182) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$restoreShard$1(BlobStoreRepository.java:4291) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.3.2.jar:3.3.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: org.opensearch.index.shard.IndexShardRecoveryException: failed recovery
	... 17 more
Caused by: org.opensearch.index.snapshots.IndexShardRestoreFailedException: restore failed
	... 15 more
Caused by: org.opensearch.index.snapshots.IndexShardRestoreFailedException: failed to restore snapshot [1/1jJfc8QJQxSFRMgnopbSfg]
	... 13 more
Caused by: java.lang.IllegalArgumentException: can't add field [__nested_parent] as parent document field; this IndexWriter has no parent document field configured
	at org.apache.lucene.index.FieldInfos$FieldNumbers.verifyParentFieldName(FieldInfos.java:489) ~[lucene-core-10.3.1.jar:10.3.1 51190f35a16d2ce433139abfe0fd8365791b352a - 2025-10-02 09:50:16]
	at org.apache.lucene.index.FieldInfos$FieldNumbers.addOrGet(FieldInfos.java:420) ~[lucene-core-10.3.1.jar:10.3.1 51190f35a16d2ce433139abfe0fd8365791b352a - 2025-10-02 09:50:16]
	at org.apache.lucene.index.IndexWriter.getFieldNumberMap(IndexWriter.java:1280) ~[lucene-core-10.3.1.jar:10.3.1 51190f35a16d2ce433139abfe0fd8365791b352a - 2025-10-02 09:50:16]
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1127) ~[lucene-core-10.3.1.jar:10.3.1 51190f35a16d2ce433139abfe0fd8365791b352a - 2025-10-02 09:50:16]
	at org.opensearch.common.lucene.Lucene.pruneUnreferencedFiles(Lucene.java:231) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.FileRestoreContext.afterRestore(FileRestoreContext.java:208) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.repositories.blobstore.FileRestoreContext.lambda$restore$0(FileRestoreContext.java:185) ~[opensearch-3.3.2.jar:3.3.2]
	at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.3.2.jar:3.3.2]

Related component

Storage:Snapshots

To Reproduce

1. Create a new snapshot repository

curl -XPUT -H 'Content-type: application/json' 'http://localhost:9200/_snapshot/test-fs-repository' -d '{
  "type": "fs",
  "settings": {
    "location": "/tmp/snapshots"
  }
}' 

2. Setup test index:

curl -XPUT  -H 'Content-type: application/json' 'http://localhost:9200/test' -d '{
  "settings": {
    "index": {
      "sort.field": "timestamp",
      "sort.order": "desc"
    }
  },
  "mappings": {
    "properties": {
      "name": {
      	"type": "text"
      },
      "timestamp": {
        "type": "date"
      }
    }
  }
}'

3. Index test document:

curl -XPOST -H 'Content-type: application/json' 'http://localhost:9200/test/_doc/1' -d '{
	"name": "Test document",
	"timestamp": "2025-11-21T15:40:00Z"
}'

4. Create a snapshot:

curl -XPUT -H 'Content-type: application/json' 'http://localhost:9200/_snapshot/test-fs-repository/1'

5. Delete the original index:

curl -XDELETE 'http://localhost:9200/test'

6. Try to restore the index:

curl -XPOST 'http://localhost:9200/_snapshot/test-fs-repository/1/_restore'

It results in the error described in the ticket description.

Expected behavior

The expected behavior is that the index is restored successfully.

Additional Details

Plugins
Clean OpenSearch installation, no additional plugins.

Screenshots
No screenshot.

Host/Environment (please complete the following information):

• OS: MacOS 26.1
• Version 3.3.2

Additional context
No additional context.

[andrross]

Catch All Triage - 1 2 3

Thank @gr0, looks like a bug. Have you attempted to reproduce this on any other versions? Is a new error in 3.3?

[cwperks]

Looks like this could be the most relevant part of the message

nested: IllegalArgumentException[can't add field [__nested_parent] as parent document field; this IndexWriter has no parent document field configured

[gr0]

My pleasure @andrross - it seems to be present in version 3.x. I quickly tested 3.0 and I got the same behavior. The same operation works in 2.x.

[msfroh]

Historically, you could not do index sorting + parent/child blockjoins (what OpenSearch calls nested documents). I remember a Lucene issue (in Lucene 10.x) that was intended to address this. Maybe it somehow broke it?

It's interesting that your mapping there doesn't have a nested field.

It looks like that __nested_parent was added in #18536. My guess is that it's getting added to the IndexWriter when the index is created, but not getting added to the IndexWriter during the snapshot restore.

Specifically, this is the IndexWriter that gets created during pruneUnreferencedFiles:

OpenSearch/server/src/main/java/org/opensearch/common/lucene/Lucene.java

Lines 225 to 234 in 332efe3

	try (
	IndexWriter writer = new IndexWriter(
	directory,
	new IndexWriterConfig(Lucene.STANDARD_ANALYZER).setSoftDeletesField(Lucene.SOFT_DELETES_FIELD)
	.setIndexCommit(cp)
	.setCommitOnClose(false)
	.setMergePolicy(NoMergePolicy.INSTANCE)
	.setOpenMode(IndexWriterConfig.OpenMode.APPEND)
	)
	) {

I bet adding .setParentField(PARENT_FIELD) to that IndexWriterConfig would fix it.

The good news is that it looks like the bug is on the restore logic, not in the snapshot itself.

[andrross]

@msfroh You're right that adding .setParentField(...) will indeed fix it. However, we cannot add the parent field if we're restoring an older index that does not have a parent field set (i.e. an OpenSearch index older than version 3.2):

can't add a parent field to an already existing index without a parent field
java.lang.IllegalArgumentException: can't add a parent field to an already existing index without a parent field
	at __randomizedtesting.SeedInfo.seed([CACAED94C2B8DEEE:6851D3FBD1D75140]:0)
	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:1132)
	at org.opensearch.common.lucene.Lucene.pruneUnreferencedFiles(Lucene.java:237)
	at org.opensearch.common.lucene.LuceneTests.testPruneUnreferencedFiles(LuceneTests.java:194)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:565)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)

Is there a way we can avoid having to know ahead of time whether these indexes include a parent field? Seems like the following might work, but is there a better way to do this?

boolean hasParentField = false;
try (DirectoryReader reader = DirectoryReader.open(directory)) {
    for (LeafReaderContext ctx : reader.leaves()) {
        FieldInfos fieldInfos = ctx.reader().getFieldInfos();
        for (FieldInfo fi : fieldInfos) {
            if (fi.isParentField()) {
                hasParentField = true;
                break;
            }
        }
    }
}
final IndexCommit cp = getIndexCommit(si, directory);
final IndexWriterConfig iwc = new IndexWriterConfig(Lucene.STANDARD_ANALYZER).setSoftDeletesField(Lucene.SOFT_DELETES_FIELD)
    .setIndexCommit(cp)
    .setCommitOnClose(false)
    .setMergePolicy(NoMergePolicy.INSTANCE)
    .setOpenMode(IndexWriterConfig.OpenMode.APPEND);
if (hasParentField) {
    iwc.setParentField(PARENT_FIELD);
}
try (IndexWriter writer = new IndexWriter(directory, iwc)) {
    // do nothing and close this will kick off IndexFileDeleter which will remove all pending files
}
return si;

[andrross]

FYI @bharath-techie & @vishdivs

[msfroh]

@andrross -- that's actually a pretty good way of doing it. In theory, we could even set the parent field in the writer to whatever field we find in the FieldInfos (though for now, we only set it to that one constant).

Given that this scenario was clearly not tested, I'm wondering how it works once the restore completes. We must open another IndexWriter to process new writes to the index, unless we restore it readonly. Do we check for a parent field when we open that writer? Maybe we do, because if we didn't, it would break rolling upgrades.

[mgodwan]

Do we check for a parent field when we open that writer? Maybe we do, because if we didn't, it would break rolling upgrades.

Other recovery flows seem to be covered through the following IndexWriterConfig creation flows:

OpenSearch/server/src/main/java/org/opensearch/index/store/Store.java

Lines 1981 to 1983 in f31300f

	if (this.isIndexSortEnabled && this.isParentFieldEnabledVersion) {
	iwc.setParentField(Lucene.PARENT_FIELD);
	}

OpenSearch/server/src/main/java/org/opensearch/index/engine/NativeLuceneIndexWriterFactory.java

Lines 205 to 210 in f31300f

	if (engineConfig.getIndexSort() != null) {
	iwc.setIndexSort(engineConfig.getIndexSort());
	if (engineConfig.getIndexSettings().getIndexVersionCreated().onOrAfter(Version.V_3_2_0)) {
	iwc.setParentField(Lucene.PARENT_FIELD);
	}
	}

We can possibly leverage the members in store to decide whether parent field should be set or not.