Skip to content

S3 attachment storage#1656

Merged
binwiederhier merged 37 commits into
mainfrom
attachment-s3
Mar 23, 2026
Merged

S3 attachment storage#1656
binwiederhier merged 37 commits into
mainfrom
attachment-s3

Conversation

@binwiederhier
Copy link
Copy Markdown
Owner

@binwiederhier binwiederhier commented Mar 15, 2026

Summary

Adds S3-compatible object storage as a backend for attachment caching, as an alternative to local filesystem storage.

  • Minimal S3 client (s3/ package): implements PutObject (with automatic multipart upload for large files), GetObject, DeleteObjects, and ListObjectsV2 using AWS Signature V4 signing — no AWS SDK dependency
  • Attachment store (attachment/ package): unified Store with pluggable backend interface (fileBackend / s3Backend), shared size tracking, rate limiting, and background sync
  • Background cleanup: every 15 minutes, reconciles the S3 bucket (or prefix) with the message database — deletes orphaned objects whose file IDs are no longer referenced, and aborts incomplete multipart uploads older than 1 hour
  • Configuration: attachment-cache-dir accepts an S3 URL (s3://ACCESS_KEY:SECRET_KEY@BUCKET[/PREFIX]?region=REGION[&endpoint=ENDPOINT]), auto-detecting path-style addressing when a custom endpoint is set
  • Documentation: updated docs/config.md with S3 configuration examples and cleanup behavior

S3 client features

  • AWS Signature V4 request signing (no external dependencies)
  • Simple PUT for small objects, automatic multipart upload (5 MB parts) for large objects
  • Batch delete (up to 1000 keys per call)
  • Paginated ListObjectsV2 with prefix filtering
  • ListMultipartUploads + AbortMultipartUpload for cleaning up incomplete uploads
  • Path-style and virtual-hosted-style addressing
  • Comprehensive test suite with mock S3 server

Test plan

  • go test ./s3/... — S3 client unit tests (signing, URL construction, prefix handling) and integration tests (mock S3 server)
  • go test ./attachment/... — attachment store tests for both file and S3 backends
  • go test ./server/... — server tests pass
  • go test ./util/... — LimitReader, CountingReader, and LimitWriter tests

@binwiederhier binwiederhier changed the title WIP: S3 S3 attachment storage Mar 21, 2026
@binwiederhier binwiederhier merged commit 2f6a044 into main Mar 23, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant