[META] Remote Store Issue List

[sachinpkale]

Is your feature request related to a problem? Please describe.

• Issues listed under this meta issue are related to Remote Segment Store.

Describe the solution you'd like

• Feature Proposal - [Feature Proposal] Add Remote Storage Options for Improved Durability #1968
• Design Proposal - [Design Proposal] Add Remote Storage Options for Improved Durability #2700

Tasks

OpenSearch 2.3 Release

• [Remote Segment Store] Define RemoteDirectory interface #2993
• [Remote Store] Add index level setting to enable/disable remote store #3141
• [Remote Segment Store] Add another instance of Store to InternalEngine that uses RemoteDirectory #2994
• [Remote Segment Store] Upload new segment files to remote store as a part of refresh flow #2995
• [Remote Store] API support to restore data from remote store #3145
• [Remote Store] Add a check to enable remote store only with Segment Replication #3147
• [Remote Store] Performance test setup for 2.3 release #3148
• [Remote Store] Make repository name configurable #3286
• [Remote Translog] Add extension points to support remote translog stores #3242
• [Remote Segment Store] Upload segments_N file after each refresh #3398
• [Remote Store] Move repository fetch logic from IndexService to RemoteDirectoryFactory #3547
• [Remote Store] Inject the RemoteStoreRefreshListener in the indexShard constructor #3548
• [Remote Store] Support nested index level settings for remote store #4059
• [Remote Store] Fix message on enabling remote store without segment replication #4201
• [BUG] [Remote Store] AssertionError while creating an index with index.remote_store.enabled=true #4233
• [BUG][Remote Store] Repository base_path is ignored while uploading segments to remote segment store #4283
• [BUG][Remote Segment Store] On failover, new primary uploads all the segments to remote store, it should be incremental #4398
• [Remote Segment Store] Delete merged away segment files from remote segment store #3142
• [BUG][Remote Store] Repository base_path is ignored while uploading segments to remote segment store #4283

Backlog

• [Remote Segment Store] Validate checksum of uploaded and downloaded segment files #3143
• [Remote Store] Add support to auto restore data from remote store #3144
• [Remote Store] Add API to check status of remote store sync #3146
• [Discuss] [Remote Store] Supporting multiple remote stores per index #4437
• [Remote Translog] Integrate remote translog store with recovery #3255
• [Remote Store] Handle transient and permanent un-availability of the remote store #3397
• [Remote Store] Upload index metadata to remote store #3457
• [Remote Store] Delete data from remote store (translog and segments) on index delete #3511
• [Remote Store] Make remote segment store and remote translog path consistent #3589
• [Remote Store] Add wrapper for RestHighLevelClient that provides methods for accessing the Remote Store API. #3602
• [FEATURE] Add java client support for remote store APIs opensearch-java#168
• Add OpenSearch documentation for remotestore APIs and mark the APIs stable #3667
• [Remote Store] Create abstraction CompositeStore which will encapsulate local and remote store #3719
• [Pluggable Translog] Decouple Translog from Engine #3241
• [META] Provide extension points for Remote Translog  #3600
• Remote Translog plugin implementation for S3 #3601
• [Discuss] Writes on NRT Replica with Remote Translog  #3706
• Move Translog dependency from core to a module #3599
• [Remote Store] Integration of remote translog and remote segment store #3754
• [Remote Store] Integration test scenarios #3759
• [Remote Store] Restore API to support shard level restore #3768
• [Remote Store] Experimental Release Tracking #3974
• [Remote Store] Make uploadMetadata private and call it internally from RemoteSegmentStoreDirectory #4171
• [Remote Store] Create abstraction for RemoteSegmentStoreDirectory.init() method #4410
• [Remote Store] Add support for default repository name for remote store #4409
• [Remote Store] Fail index creation with error message if repository does not exist #4403
• [Remote Store] Support bulk upload and bulk delete while interacting with remote segment store #4317
• [Remote Store] Inject individual dependencies rather than injecting IndexShard into RemoteStoreRefreshListener #4316
• [Remote Store] Delete stale commits in async flow #4315
• [Remote Store] Track segment uploads as a part of an upload listener. #4198
• [Remote Store] Stop creating translog generation and checkpoint files on replicas where translog is stored remotely #4232
• [Remote Store] Introduce segment replication as prefinal step during recovery of replicas  #4231
• [Remote Store] Skip translog replay in Recovery code for the replica's recovery when translogs stored remotely.  #4230
• [Remote Store] With no-op replication, remove the payload to the in-sync calls to replicas #4229
• [Remote Store] Keep metadata files for last N refreshes #5455
• [Remote Store] Add upload timestamp to metada file to be used in conflict resolution #5475