Restore snapshot changes for shallow snapshot V2#15462

Merged

sachinpkale merged 6 commits intoopensearch-project:mainfrom

sachinpkale:v2-snapshot-restore

Sep 3, 2024

Member

sachinpkale commented Aug 28, 2024 •

edited

Loading

Description

As part of shallow snapshot V2 optimizations, we introduced timestamp pinning functionality for remote store data.
This PR contains changes related to restore of shallow snapshot V2.
Restoring shallow snapshot V2 involves following changes:
- Identify if snapshot is shallow snapshot V2 (based on pinned timestamp field in snapshot info)
- Restore segments corresponding to the pinned timestamp in snapshot
- Restore and replay translog corresponding to the pinned timestamp in snapshot

Related Issues

[Timestamp Pinning in Remote Store] Changes in remote store restore flow to restore data at given timestamp #15066

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Contributor

github-actions bot commented Aug 28, 2024

❌ Gradle check result for c9d6deb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from c9d6deb to e76753f Compare

August 28, 2024 09:35

Contributor

github-actions bot commented Aug 28, 2024

❌ Gradle check result for e76753f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch 2 times, most recently from a5dd779 to 56cca2a Compare

August 28, 2024 12:03

Contributor

github-actions bot commented Aug 28, 2024

❌ Gradle check result for a5dd779: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented Aug 28, 2024

❌ Gradle check result for 56cca2a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from 56cca2a to 58666c2 Compare

August 29, 2024 09:45

Contributor

github-actions bot commented Aug 29, 2024

❌ Gradle check result for 58666c2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch 2 times, most recently from b16d343 to df1b7ac Compare

August 30, 2024 04:16

Contributor

github-actions bot commented Aug 30, 2024

❌ Gradle check result for b16d343: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Contributor

github-actions bot commented Aug 30, 2024

❌ Gradle check result for df1b7ac: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from df1b7ac to 570d138 Compare

August 31, 2024 09:59

Contributor

github-actions bot commented Aug 31, 2024

❌ Gradle check result for 570d138: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from 570d138 to ea6c2d3 Compare

September 1, 2024 08:20

Contributor

github-actions bot commented Sep 1, 2024

❌ Gradle check result for ea6c2d3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from ea6c2d3 to 8d49a6f Compare

September 1, 2024 09:06

Contributor

github-actions bot commented Sep 1, 2024

❌ Gradle check result for 8d49a6f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from 8d49a6f to 2fc3b51 Compare

September 1, 2024 16:12

Contributor

github-actions bot commented Sep 1, 2024

❌ Gradle check result for 2fc3b51: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch from 2fc3b51 to 4a68ff5 Compare

September 1, 2024 16:45

sachinpkale added the skip-changelog label

Contributor

github-actions bot commented Sep 1, 2024

❌ Gradle check result for 4a68ff5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale force-pushed the v2-snapshot-restore branch 2 times, most recently from a2fa7e6 to 58cd123 Compare

September 2, 2024 04:04

Contributor

github-actions bot commented Sep 2, 2024

❌ Gradle check result for a2fa7e6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sachinpkale changed the title ~~Restore snapshot changes for V2~~ Restore snapshot changes for shallow snapshot V2

sachinpkale marked this pull request as ready for review

September 2, 2024 04:12

sachinpkale requested a review from anasalkouz as a code owner

September 2, 2024 04:12

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

d524515

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

8fc0d8a

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added a commit to sachinpkale/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

1c12a6f

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>

sachinpkale added a commit that referenced this pull request


          Flaky tests fix in 2.x branch (#15805)

a24c858

* Mute test testRemoteStoreCleanupForDeletedIndexForSnapshotV2 (#15695)

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>

* Mute flaky test RemoteFsTimestampAwareTranslogTests.testSimpleOperationsUpload (#15732)

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>

* Mute failing snapshot status tests (#15652)

Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Lakshya Taragi <157457166+ltaragi@users.noreply.github.com>

* Restore snapshot changes for shallow snapshot V2 (#15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>

* version change to V_2.17.0 for node min version check (#15644)

Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>

---------

Signed-off-by: Marc Handalian <marc.handalian@gmail.com>
Signed-off-by: Sachin Kale <kalsac@amazon.com>
Signed-off-by: Lakshya Taragi <lakshya.taragi@gmail.com>
Signed-off-by: Lakshya Taragi <157457166+ltaragi@users.noreply.github.com>
Signed-off-by: Anshu Agarwal <anshukag@amazon.com>
Co-authored-by: Marc Handalian <marc.handalian@gmail.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Lakshya Taragi <157457166+ltaragi@users.noreply.github.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>
Co-authored-by: Anshu Agarwal <anshuagarwal11@gmail.com>
Co-authored-by: Anshu Agarwal <anshukag@amazon.com>

This was referenced Sep 6, 2024

[AUTOCUT] Gradle Check Flaky Test Report for DedicatedClusterSnapshotRestoreIT #15806

Open

[AUTOCUT] Gradle Check Flaky Test Report for ApproximatePointRangeQueryTests #15807

Open

[AUTOCUT] Gradle Check Flaky Test Report for MasterServiceTests #15809

Closed

[AUTOCUT] Gradle Check Flaky Test Report for PluginInfoIT #15814

Closed

[AUTOCUT] Gradle Check Flaky Test Report for SnapshotStatusApisIT #15815

Closed

[AUTOCUT] Gradle Check Flaky Test Report for DecommissionControllerTests #15833

Open

[AUTOCUT] Gradle Check Flaky Test Report for DeleteSnapshotIT #15834

Open

[AUTOCUT] Gradle Check Flaky Test Report for ReactorNetty4StreamingStressIT #15840

Closed

[AUTOCUT] Gradle Check Flaky Test Report for RepositoriesIT #15841

Closed

[AUTOCUT] Gradle Check Flaky Test Report for SearchingIT #15844

Closed

[AUTOCUT] Gradle Check Flaky Test Report for SharedClusterSnapshotRestoreIT #15845

Open

akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

776b20e

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>

This was referenced Sep 11, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MixedClusterClientYamlTestSuiteIT #14294

Open

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreStatsIT #14310

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteRestoreSnapshotIT #14324

Open

[AUTOCUT] Gradle Check Flaky Test Report for SmokeTestMultiNodeClientYamlTestSuiteIT #14408

Open

[AUTOCUT] Gradle Check Flaky Test Report for SearchRestCancellationIT #14311

Open

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

[AUTOCUT] Gradle Check Flaky Test Report for SegmentReplicationUsingRemoteStoreIT #14329

Open

prudhvigodithi mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for MixedClusterClientYamlTestSuiteIT prudhvigodithi/opensearch-build#82

Open

dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

daaa9e9

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>

dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

41e7343

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>

dk2k pushed a commit to dk2k/OpenSearch that referenced this pull request


          Restore snapshot changes for shallow snapshot V2 (opensearch-project#…

721289b

…15462)

---------

Signed-off-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Sachin Kale <kalsac@amazon.com>
Co-authored-by: Gaurav Bafna <gbbafna@amazon.com>

opensearch-ci-bot mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for RestoreShallowSnapshotV2IT #16658

Open

opensearch-ci-bot mentioned this pull request

[AUTOCUT] Gradle Check Flaky Test Report for ConcurrentSnapshotsIT #20107

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

gbbafna gbbafna approved these changes

anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

andrross Awaiting requested review from andrross andrross is a code owner

ashking94 Awaiting requested review from ashking94 ashking94 is a code owner

Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

dblock Awaiting requested review from dblock

dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

jed326 Awaiting requested review from jed326 jed326 is a code owner

kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

mch2 Awaiting requested review from mch2 mch2 is a code owner

msfroh Awaiting requested review from msfroh msfroh is a code owner

nknize Awaiting requested review from nknize

owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

reta Awaiting requested review from reta reta is a code owner

Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

sohami Awaiting requested review from sohami sohami is a code owner

VachaShah Awaiting requested review from VachaShah

jainankitk Awaiting requested review from jainankitk

linuxpi Awaiting requested review from linuxpi

Labels

backport 2.x backport-failed skip-changelog