[Enhancement] Migration from sonatype snapshots repo to ci.opensearch.org snapshots repo

[peterzhuamazon]

[Enhancement] Migration from sonatype snapshots repo to ci.opensearch.org snapshots repo

---

Sonatype is having new policies of removing snapshots over 30 days.

Steps:

1. Copy existing snapshots from https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/ to https://ci.opensearch.org/ci/dbc/snapshots/org/opensearch/
2. Update all OS core + plugins + standalone repo to publish snapshots to new location
3. Update all OS core + plugins + standalone repo to pull snapshots from new location

Additional request: browsing to make sure s3 is online

Thanks.

---

PRs:

• opensearch-build: publish-snapshot.sh ✅
  • Support s3 bucket in publish-snapshot.sh script #5789
• OpenSearch (core) ✅
  • Onboarding new maven snapshots publishing to s3 (OpenSearch Core) OpenSearch#19619
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (OpenSearch Core) OpenSearch#19632
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (OpenSearch Core) OpenSearch#19668
• notifications ✅
  • Onboarding new maven snapshots publishing to s3 (notifications) notifications#1088
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (notifications) notifications#1089
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (notifications) notifications#1091
• geospatial ✅
  • Onboarding new maven snapshots publishing to s3 (geospatial) geospatial#804
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (geospatial) geospatial#805
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (geospatial) geospatial#807
• search-relevance ✅ (no 2.19)
  • Onboarding new maven snapshots publishing to s3 (Search Relevance) search-relevance#269
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (Search Relevance) search-relevance#280
• opensearch-learning-to-rank-base ✅
  • Onboarding new maven snapshots publishing to s3 (LTR) opensearch-learning-to-rank-base#239
  • Onboarding new maven snapshots publishing to s3 and bump version opensearch-learning-to-rank-base#240
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (LTR) opensearch-learning-to-rank-base#243
• observability ✅
  • Onboarding new maven snapshots publishing to s3 (observabillity) observability#1953
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (observabillity) observability#1954
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (observabillity) observability#1956
• opensearch-system-templates ✅
  • Onboarding new maven snapshots publishing to s3 (system-templates) opensearch-system-templates#104
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (system-templates) opensearch-system-templates#105
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (system-templates) opensearch-system-templates#107
• job-scheduler ✅
  • Onboarding new maven snapshots publishing to s3 (job-scheduler) job-scheduler#850
    • Resolve #850 issues job-scheduler#851
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (job-scheduler) job-scheduler#854
    • [Backport 3.3] Resolve #850 issues job-scheduler#855
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (job-scheduler) job-scheduler#852
    • [Backport 2.19] Resolve #850 issues job-scheduler#853
  • Remove cron trigger on JS as maven snapshots is already on s3 job-scheduler#858
• reporting ✅
  • Onboarding new maven snapshots publishing to s3 (reporting) reporting#1132
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (reporting) reporting#1134
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (reporting) reporting#1136
• common-utils ✅
  • Onboarding new maven snapshots publishing to s3 (common-utils) common-utils#884
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (common-utils) common-utils#885
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (common-utils) common-utils#886
• asynchronous-search ✅
  • Onboarding new maven snapshots publishing to s3 (async) asynchronous-search#782
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (async) asynchronous-search#783
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (async) asynchronous-search#785
• security-analytics ✅
  • Onboarding new maven snapshots publishing to s3 (SA) security-analytics#1597
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (SA) security-analytics#1600
  • Bump maven snapshot publishing to java 21 security-analytics#1601
  • [Backport 2.19] Bump maven snapshot publishing to java 21 security-analytics#1602
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (SA) security-analytics#1603
  • [Backport 3.3] Bump maven snapshot publishing to java 21 security-analytics#1604
• skills ✅
  • Onboarding new maven snapshots publishing to s3 (skills) skills#657
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (skills) skills#659
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (skills) skills#662
• alerting ✅
  • Onboarding new maven snapshots publishing to s3 (alerting) alerting#1949
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (alerting) alerting#1950
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (alerting) alerting#1959
• k-NN ✅
  • Onboard to s3 snapshots k-NN#2943
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (k-NN) k-NN#2945
  • [Backport 3.3] Onboard to s3 snapshots k-NN#2951
• anomaly-detection ✅
  • Onboarding new maven snapshots publishing to s3 (AD) anomaly-detection#1590
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (AD) anomaly-detection#1592
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (AD) anomaly-detection#1593
• user-behavior-insights ✅ (no 2.19)
  • Onboarding new maven snapshots publishing to s3 (UBI) user-behavior-insights#140
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (UBI) user-behavior-insights#143
• neural-search ✅
  • Onboard to s3 snapshots neural-search#1618
  • [Backport 2.19] Onboard to s3 snapshots neural-search#1622
  • [Backport 3.3] Onboard to s3 snapshots neural-search#1625
• performance-analyzer ✅
  • Onboarding new maven snapshots publishing to s3 (PA) performance-analyzer#853
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (PA) performance-analyzer#854
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (PA) performance-analyzer#858
• performance-analyzer-commons ✅ (no 2.19)
  • Onboarding new maven snapshots publishing to s3 (PA-commons) performance-analyzer-commons#111
• performance-analyzer-rca (2.x only, main old)
  • Onboarding new maven snapshots publishing to s3 (PA-RCA) performance-analyzer-rca#602
• opensearch-remote-metadata-sdk ✅
  • Onboarding new maven snapshots publishing to s3 (opensearch-remote-metadata-sdk) opensearch-remote-metadata-sdk#267
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (opensearch-remote-metadata-sdk) opensearch-remote-metadata-sdk#268
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (opensearch-remote-metadata-sdk) opensearch-remote-metadata-sdk#269
• index-management ✅
  • Onboarding new maven snapshots publishing to s3 (IM) index-management#1513
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (IM) index-management#1514
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (IM) index-management#1519
• custom-codecs ✅
  • Onboarding new maven snapshots publishing to s3 (custom-codecs) custom-codecs#287
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (custom-codecs) custom-codecs#288
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (custom-codecs) custom-codecs#290
• flow-framework ✅
  • Onboarding new maven snapshots publishing to s3 (flow-framework) flow-framework#1255
  • Onboarding new maven snapshots publishing to s3 and bump version flow-framework#1258
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (flow-framework) flow-framework#1260
• sql (SNAPSHOT_REPO_URL) ✅
  • Onboarding new maven snapshots publishing to s3 (sql) sql#4588
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (sql) sql#4589
  • Onboarding async query core and grammar files to maven snapshots sql#4598
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (sql) sql#4639
• cross-cluster-replication ✅
  • Onboarding new maven snapshots publishing to s3 (CCR) cross-cluster-replication#1592
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (CCR) cross-cluster-replication#1594
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (CCR) cross-cluster-replication#1597
• ml-commons ✅
  • Onboard to s3 snapshots ml-commons#4320
  • [Backport 2.19] Onboard to s3 snapshots ml-commons#4322
  • [Backport 3.1] Onboard to s3 snapshots ml-commons#4327
  • Use regex to publish maven snapshots based on branches ml-commons#4330
  • [Backport 3.2] Onboard to s3 snapshots ml-commons#4329
  • [Backport 3.3] Onboard to s3 snapshots ml-commons#4328
• security: ✅
  • Onboarding new maven snapshots publishing to s3 (security) security#5711
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (security) security#5716
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (security) security#5719
• query-insights ✅
  • Onboarding new maven snapshots publishing to s3 (query-insights) query-insights#467
  • [Backport 2.19] Onboarding new maven snapshots publishing to s3 (query-insights) query-insights#469
  • [Backport 3.3] Onboarding new maven snapshots publishing to s3 (query-insights) query-insights#471
• jVector ✅ (no 2.19)
  • Onboarding new maven snapshots publishing to s3 (jVector) opensearch-jvector#178
• protobufs: ✅ (no 2.19)
  • Onboarding new maven snapshots publishing to s3 (protobufs) opensearch-protobufs#262
• New plugins ✅ (no 2.19)

---

• opensearch-plugin-template-java ✅
  • Onboarding new maven snapshots publishing to s3 (plugin template java) opensearch-plugin-template-java#112
• opensearch-java: ✅
  • Onboarding new maven snapshots publishing to s3 (opensearch-java) opensearch-java#1768
  • [Backport 3.x] Onboarding new maven snapshots publishing to s3 (opensearch-java) opensearch-java#1774
  • Pending 2.x
• spring-data-opensearch: ✅
  • Onboarding new maven snapshots publishing to s3 (spring-data-opensearch) spring-data-opensearch#569
• opensearch-spark (SNAPSHOT_REPO_URL): ⚠️
  • Spark is using a very specific method of publishing which does not support s3 publishing, until they migrate to be the same as sql repo method.
  • [Enhancement] Onboard to the same maven snapshots publishing method as sql to support s3 snapshots opensearch-spark#1282
• data-prepper: ⏳
  • Onboarding new maven snapshots publishing to s3 (data-prepper) data-prepper#6246
• opensearch-testcontainers: ✅
  • Onboarding new maven snapshots publishing to s3 (testcontainers) opensearch-testcontainers#306
  • Fix snapshot publication opensearch-testcontainers#307
• opensearch-hadoop (SNAPSHOT_REPO_URL): ✅
  • Onboarding new maven snapshots publishing to s3 (opensearch-hadoop) opensearch-hadoop#644
• opensearch-prometheus-exporter: ✅
  • Onboarding new maven snapshots publishing to s3 (promethueus exporter) opensearch-prometheus-exporter#415
• cluster-etcd: ✅
  • Onboarding new maven snapshots publishing to s3 (cluster-etcd) cluster-etcd#62
• opensearch-storage-encryption: ✅
  • Onboarding new maven snapshots publishing to s3 (storage-encryption) opensearch-storage-encryption#76
• opensearch-api-specification: ✅
  • Onboarding new maven snapshots publishing to s3 (api-specification) opensearch-api-specification#985
• opensearch-sdk-java: ⚠️
  • sdk java havent even support OS3 as of now
  • Onboarding new maven snapshots publishing to s3 (opensearch-sdk-java) opensearch-sdk-java#1034

[getsaurabh02]

+1 @peterzhuamazon. We should move our snapshot CI building to an S3 buckets instead

[peterzhuamazon]

Start migration of 40G worth of data to s3.
2025/03/05 11:19 PST

[peterzhuamazon]

Since Sonatype does not have previous deleted snapshots, we will individually build them if needed.
Complete migration of existing snapshots: 43G.

Thanks.