Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/features/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@
- [Refresh Task Scheduling](opensearch/refresh-task-scheduling.md)
- [Replication](opensearch/replication.md)
- [Remote Store Metrics](opensearch/remote-store-metrics.md)
- [S3 Repository](opensearch/s3-repository.md)
- [Search Backpressure](opensearch/search-backpressure.md)
- [Search Request Stats](opensearch/search-request-stats.md)
- [Secure Transport Settings](opensearch/secure-transport-settings.md)
Expand Down
167 changes: 167 additions & 0 deletions docs/features/opensearch/s3-repository.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
# S3 Repository

## Summary

The S3 Repository plugin (`repository-s3`) enables OpenSearch to store snapshots in Amazon S3 buckets. It provides a reliable, scalable, and cost-effective solution for backing up and restoring OpenSearch indices using AWS cloud storage.

## Details

### Architecture

```mermaid
graph TB
subgraph "OpenSearch Cluster"
OS[OpenSearch Node]
Plugin[repository-s3 Plugin]
OS --> Plugin
end

subgraph "AWS"
S3[Amazon S3 Bucket]
IAM[IAM Credentials]
end

Plugin -->|Snapshot/Restore| S3
Plugin -->|Authentication| IAM

subgraph "Retry Mechanism"
Retry[Standard Retry Mode]
Backoff[Jittered Exponential Backoff]
Circuit[Circuit Breaker]
Retry --> Backoff
Backoff --> Circuit
end

Plugin --> Retry
```

### Components

| Component | Description |
|-----------|-------------|
| `S3Service` | Manages synchronous S3 client operations |
| `S3AsyncService` | Manages asynchronous S3 client operations |
| `S3Repository` | Implements the repository interface for S3 storage |
| `S3BlobStore` | Handles blob storage operations in S3 |
| `S3BlobContainer` | Manages blob containers within S3 |

### Configuration

#### Repository Settings

| Setting | Description | Default |
|---------|-------------|---------|
| `bucket` | Name of the S3 bucket | Required |
| `base_path` | Path within the bucket for snapshots | Root |
| `client` | Named client configuration to use | `default` |
| `compress` | Whether to compress metadata files | `false` |
| `chunk_size` | Size of file chunks for uploads | `1gb` |
| `max_retries` | Maximum number of retry attempts | `3` |
| `throttle_retries` | Use throttling backoff strategy | `true` |
| `canned_acl` | S3 canned ACL for created objects | `private` |
| `storage_class` | S3 storage class for snapshot files | `standard` |
| `server_side_encryption` | Enable S3 server-side encryption | `false` |
| `buffer_size` | Buffer size for multipart uploads | `5mb`-`5gb` |
| `max_restore_bytes_per_sec` | Maximum restore rate | `40mb` |
| `max_snapshot_bytes_per_sec` | Maximum snapshot rate | `40mb` |
| `readonly` | Whether repository is read-only | `false` |

#### Client Settings

Configure in `opensearch.yml`:

| Setting | Description |
|---------|-------------|
| `s3.client.default.access_key` | AWS access key |
| `s3.client.default.secret_key` | AWS secret key |
| `s3.client.default.endpoint` | Custom S3 endpoint |
| `s3.client.default.region` | AWS region |
| `s3.client.default.protocol` | HTTP or HTTPS |

### Usage Example

#### Register S3 Repository

```json
PUT _snapshot/my-s3-repo
{
"type": "s3",
"settings": {
"bucket": "my-opensearch-snapshots",
"base_path": "snapshots/production",
"compress": true,
"server_side_encryption": true
}
}
```

#### Create Snapshot

```json
PUT _snapshot/my-s3-repo/snapshot-1
{
"indices": "my-index-*",
"ignore_unavailable": true,
"include_global_state": false
}
```

#### Restore Snapshot

```json
POST _snapshot/my-s3-repo/snapshot-1/_restore
{
"indices": "my-index-*",
"rename_pattern": "(.+)",
"rename_replacement": "restored-$1"
}
```

### Retry Behavior

The plugin uses AWS SDK's Standard retry mode (since v2.18.0):

- **Maximum Attempts**: 3 by default
- **Backoff Strategy**: Jittered exponential backoff
- **Circuit Breaker**: Prevents retries during outages
- **Throttling**: Automatic handling of S3 throttling responses

```mermaid
flowchart TB
A[Request] --> B{Success?}
B -->|Yes| C[Complete]
B -->|No| D{Retryable?}
D -->|No| E[Fail]
D -->|Yes| F{Attempts < Max?}
F -->|No| E
F -->|Yes| G{Retry Quota Available?}
G -->|No| E
G -->|Yes| H[Exponential Backoff with Jitter]
H --> I[Retry Request]
I --> B
```

## Limitations

- Glacier and Deep Archive storage classes are not supported
- Maximum chunk size is limited by S3 multipart upload limits
- IAM credentials must have appropriate S3 permissions
- Cross-region snapshot restore may incur data transfer costs

## Related PRs

| Version | PR | Description |
|---------|-----|-------------|
| v2.18.0 | [#15978](https://github.com/opensearch-project/OpenSearch/pull/15978) | Change default retry mechanism to Standard Mode |
| v2.18.0 | [#16194](https://github.com/opensearch-project/OpenSearch/pull/16194) | Fix SLF4J warnings on startup |

## References

- [Issue #15397](https://github.com/opensearch-project/OpenSearch/issues/15397): Add jitter to downloads from remote store
- [Issue #16152](https://github.com/opensearch-project/OpenSearch/issues/16152): SLF4J warnings when adding repository-s3
- [Register Snapshot Repository](https://docs.opensearch.org/2.18/api-reference/snapshots/create-repository/): OpenSearch documentation
- [AWS SDK Retry Behavior](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html): AWS retry modes documentation

## Change History

- **v2.18.0** (2024-10-22): Changed default retry mechanism to Standard Mode, fixed SLF4J warnings
120 changes: 120 additions & 0 deletions docs/releases/v2.18.0/features/opensearch/s3-repository.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
# S3 Repository

## Summary

OpenSearch v2.18.0 improves the S3 repository plugin with two key changes: switching the default retry mechanism from Legacy to Standard Mode for better reliability, and fixing SLF4J logging warnings that appeared on startup when the repository-s3 plugin was installed.

## Details

### What's New in v2.18.0

#### Standard Retry Mode for S3 Clients

The S3 repository plugin now uses AWS SDK's Standard retry mode instead of the Legacy mode. Standard mode provides:

- A recommended set of retry rules across AWS SDKs
- Automatic adjustment of retry counts to maximize availability and stability
- Safe usage in multi-tenant applications
- Jittered exponential backoff for failed requests
- Circuit-breaking to prevent retries during outages

The default maximum number of attempts with Standard mode is three, unless explicitly configured.

#### SLF4J Warning Fix

Fixed startup warnings that appeared when the repository-s3 plugin was installed:

```
[WARN ][stderr] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[WARN ][stderr] SLF4J: Defaulting to no-operation (NOP) logger implementation
[WARN ][stderr] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
```

This was resolved by adding the `log4j-slf4j-impl` runtime dependency to properly bridge SLF4J logging to Log4j.

### Technical Changes

#### Retry Strategy Changes

```mermaid
graph TB
subgraph "Before v2.18.0 (Legacy Mode)"
A1[Request] --> B1[Failure]
B1 --> C1[Legacy Backoff Strategy]
C1 --> D1[Retry with SDK-specific behavior]
end

subgraph "v2.18.0+ (Standard Mode)"
A2[Request] --> B2[Failure]
B2 --> C2[Standard Backoff Strategy]
C2 --> D2[Jittered Exponential Backoff]
D2 --> E2[Circuit Breaker Check]
E2 --> F2[Retry with max 3 attempts]
end
```

#### Code Changes

| File | Change |
|------|--------|
| `S3Service.java` | Added `RetryMode.STANDARD` to throttling backoff strategy |
| `S3AsyncService.java` | Added `RetryMode.STANDARD` to throttling backoff strategy |
| `build.gradle` | Added `log4j-slf4j-impl` runtime dependency |

#### New Dependencies

| Dependency | Version | Purpose |
|------------|---------|---------|
| `log4j-slf4j-impl` | 2.21.0 | SLF4J to Log4j bridge |

### Configuration

No new configuration options were added. The retry behavior change is automatic and uses the existing `max_retries` and `throttle_retries` settings:

| Setting | Description | Default |
|---------|-------------|---------|
| `max_retries` | Maximum number of retry attempts | 3 |
| `throttle_retries` | Whether to use throttling backoff strategy | `true` |

### Usage Example

```json
PUT _snapshot/my-s3-repo
{
"type": "s3",
"settings": {
"bucket": "my-bucket",
"base_path": "snapshots",
"max_retries": 3
}
}
```

### Migration Notes

- No action required - the retry mode change is automatic
- Existing S3 repository configurations continue to work
- Users may notice improved reliability during transient S3 failures

## Limitations

- The Standard retry mode has a default maximum of 3 attempts
- Adaptive retry mode is not supported (only Standard and Legacy)

## Related PRs

| PR | Description |
|----|-------------|
| [#15978](https://github.com/opensearch-project/OpenSearch/pull/15978) | Change default retry mechanism of S3 clients to Standard Mode |
| [#16194](https://github.com/opensearch-project/OpenSearch/pull/16194) | Fix warnings from SLF4J on startup when repository-s3 is installed |

## References

- [Issue #15397](https://github.com/opensearch-project/OpenSearch/issues/15397): Add jitter to downloads from remote store
- [Issue #16152](https://github.com/opensearch-project/OpenSearch/issues/16152): SLF4J warnings when adding repository-s3
- [AWS SDK Retry Behavior](https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html): AWS documentation on retry modes
- [Register Snapshot Repository](https://docs.opensearch.org/2.18/api-reference/snapshots/create-repository/): OpenSearch S3 repository documentation

## Related Feature Report

- [Full feature documentation](../../../../features/opensearch/s3-repository.md)
1 change: 1 addition & 0 deletions docs/releases/v2.18.0/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ This page contains feature reports for OpenSearch v2.18.0.
- [Docker Compose v2 Support](features/opensearch/docker-compose-v2-support.md) - Add support for Docker Compose v2 in TestFixturesPlugin for modern Docker installations
- [Snapshot Restore Enhancements](features/opensearch/snapshot-restore-enhancements.md) - Alias renaming during restore and clone operation optimization for doc-rep clusters
- [Remote Store Metrics](features/opensearch/remote-store-metrics.md) - New REMOTE_STORE metric in Node Stats API for monitoring pinned timestamp fetch operations
- [S3 Repository](features/opensearch/s3-repository.md) - Standard retry mode for S3 clients and SLF4J warning fix

### OpenSearch Dashboards

Expand Down