[CI] FullClusterRestartIT testSingleDoc {cluster=UPGRADED} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #8617 / 8.15.0_bwc-snapshots
• elasticsearch-intake #8617 / 8.14.4_bwc-snapshots
• elasticsearch-pull-request #27244 / 7.17.23_bwc-snapshots
• elasticsearch-intake #8617 / 8.16.0_bwc-snapshots
• elasticsearch-intake #8617 / 7.17.23_bwc-snapshots
• elasticsearch-intake #8616 / 8.14.4_bwc-snapshots
• elasticsearch-intake #8616 / 8.15.0_bwc-snapshots
• elasticsearch-intake #8616 / 8.16.0_bwc-snapshots
• elasticsearch-intake #8616 / 7.17.23_bwc-snapshots

Reproduction Line:

./gradlew ":x-pack:qa:full-cluster-restart:v8.15.0#bwcTest" -Dtests.class="org.elasticsearch.xpack.restart.FullClusterRestartIT" -Dtests.method="testSingleDoc {cluster=UPGRADED}" -Dtests.seed=AD9878366FDA34F -Dtests.bwc=true -Dtests.locale=fi -Dtests.timezone=America/St_Kitts -Druntime.java=22

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:33969], URI [/testsingledoc/_doc/1], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [testsingledoc]","resource.type":"index_or_alias","resource.id":"testsingledoc","index_uuid":"_na_","index":"testsingledoc"}],"type":"index_not_found_exception","reason":"no such index [testsingledoc]","resource.type":"index_or_alias","resource.id":"testsingledoc","index_uuid":"_na_","index":"testsingledoc"},"status":404}

Issue Reasons:

• [main] 3 consecutive failures in test testSingleDoc {cluster=UPGRADED}
• [main] 2 consecutive failures in step 8.16.0_bwc-snapshots
• [main] 2 consecutive failures in step 7.17.23_bwc-snapshots
• [main] 9 failures in test testSingleDoc {cluster=UPGRADED} (1.8% fail rate in 513 executions)
• [main] 2 failures in step 8.15.0_bwc-snapshots (2.6% fail rate in 77 executions)
• [main] 2 failures in step 8.14.4_bwc-snapshots (2.6% fail rate in 77 executions)
• [main] 2 failures in step 8.16.0_bwc-snapshots (2.6% fail rate in 78 executions)
• [main] 3 failures in step 7.17.23_bwc-snapshots (3.8% fail rate in 80 executions)
• [main] 2 failures in pipeline elasticsearch-intake (7.7% fail rate in 26 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testSingleDoc {cluster=UPGRADED} (0.4% fail rate in 498 executions)

Build Scans:

• elasticsearch-intake #8616 / 8.14.4_bwc-snapshots
• elasticsearch-intake #8616 / 7.17.23_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[fang-xing-esql]

This has a similar symptom/error as #111448, index_not_found_exception, send it to data management to take a look first.

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)