Add Restore Operation to SnapshotResiliencyTests

[original-brownbear]

• Expand the successful snapshot test case to also include restoring the snapshot
  • Add indexing of documents as well to be able to meaningfully verify the restore
• This is part of the larger effort to test eventually consistent blob stores in More Snapshot Resiliency Testing #39504

---

Note: I had to add putting a mapping before indexing documents here since dynamic mapping updates trigger a blocking operation that blocks the tests. A fix for that is incoming in #39793

[elasticmachine]

Pinging @elastic/es-distributed

[original-brownbear]

Jenkins run elasticsearch-ci/2

[andrershov]

I've left two questions.
And I wonder if we want to add "restore" operation to other tests in this class as well? For example, when emulating cluster disruption?
I also want to better understand the motivation for this PR.
Is it primarily here to be able more different kind of actions in future tests?

[andrershov]

Why do we need 3 different atomic variables? Is not it enough just to have documentCountVerified?

[original-brownbear]

Right ... not sure what I was thinking :) my bad fixed in c064db3

[original-brownbear]

@andrershov The motivation for this is to basically get tests that verify that we're functional with an eventually consistent blob store (the code for that is already written in #39504) and for that we really want/need to verify that the snapshots can actually be restored. Especially in light of the fact that currently we suspect failures from eventually consistent list operations that silently corrupt the repository (#38941).
The best way to detect this kind of silent corruption imo is to simply try and restore the snapshots and check the results. With this change all we need to port is the eventually consistent mock blobstore out of #39504 and we have tests that can reproduce the corruption case :)

And I wonder if we want to add "restore" operation to other tests in this class as well? For example, when emulating cluster disruption?

Well the other ones kinda reproduce broken snapshots and we mainly cared about the fact that these snapshots finished back when we added them since the bugs resolved at the time led to various scenarios where snapshots just got stuck forever.
We could actually verify the restore in the next test too org.elasticsearch.snapshots.SnapshotResiliencyTests#testConcurrentSnapshotCreateAndDelete, but I'd do that in a subsequent step maybe?

[andrershov]

@original-brownbear so the idea is to extend testSuccessfulSnapshot with restore and call it testSuccessfulSnapshotAndRestore. And the next PR will add eventual consistent BlobStore emulation. So we can just run testSuccessfulSnapshotAndRestore for long enough time to make sure it does not fail with the eventual consistent store. Do I understand correctly, that currently, we expect it to fail or do we already have logic to work around eventual consistency?

[original-brownbear]

@andrershov

so the idea is to extend testSuccessfulSnapshot with restore and call it testSuccessfulSnapshotAndRestore.

Yea, exactly.

And the next PR will add eventual consistent BlobStore emulation. So we can just run testSuccessfulSnapshotAndRestore for long enough time to make sure it does not fail with the eventual consistent store.

Correct as well.

Do I understand correctly, that currently, we expect it to fail or do we already have logic to work around eventual consistency?

Correct as well here too. Currently the mock eventually consistent store fails yes, but I have a fix for all cases the test reproduces (tests are done too in the other PR I linked) in https://github.com/original-brownbear/elasticsearch/tree/delete-lock-via-cs (95% done, would open a PR for that once we have the tests in place).

[andrershov]

LGTM

[original-brownbear]

thanks @andrershov !