Skip to content

Bump to 3.7.0-SNAPSHOT, fix security plugin compatibility, use admin cert auth for health checks#1667

Merged
ankitkala merged 2 commits into
opensearch-project:mainfrom
cwperks:cert-auth-health-check
May 6, 2026
Merged

Bump to 3.7.0-SNAPSHOT, fix security plugin compatibility, use admin cert auth for health checks#1667
ankitkala merged 2 commits into
opensearch-project:mainfrom
cwperks:cert-auth-health-check

Conversation

@cwperks
Copy link
Copy Markdown
Member

@cwperks cwperks commented May 1, 2026

Description

Bumps OpenSearch dependency from 3.6.0-SNAPSHOT to 3.7.0-SNAPSHOT and fixes compatibility issues with the security plugin introduced by core's explicit index resolution API (PR #18523).

Changes

Version bump:

  • Updated opensearch_version default from 3.6.0-SNAPSHOT to 3.7.0-SNAPSHOT

Fix 3.7.0 security plugin compatibility:

  • Implemented TransportIndicesResolvingAction on TransportUpdateMetadataAction so the security plugin can properly resolve indices for CCR's internal metadata update action
  • Without this, the security plugin received ResolvedIndices{unknown=true} and conservatively assumed FLS/DLS/Field Masking restrictions applied, blocking all CCR replication actions with: "Cross Cluster Replication is not supported when FLS or DLS or Fieldmasking is activated"
  • Root cause: OpenSearch core PR #18523 introduced the explicit index resolution API; transport actions that don't implement it get unknown resolved indices

Admin certificate auth for health checks:

  • Replaced basic auth (admin:admin) with admin certificate auth (kirk.pem) for cluster health checks when the security plugin is enabled
  • Uses java.net.http.HttpClient which properly handles TLS client certificate presentation (unlike HttpsURLConnection)
  • Admin cert auth bypasses security index initialization, making health checks more robust during cluster startup
  • Enabled plugins.security.ssl.http.clientauth_mode: OPTIONAL so the server accepts client certificates during TLS handshake

Regenerated test certificates:

  • Regenerated esnode.pem, esnode-key.pem, kirk.pem, kirk-key.pem, and root-ca.pem with proper Subject Alternative Names including IPv6 localhost (::1), fixing hostname verification failures with java.net.http.HttpClient

Fixed admin_dn parsing:

  • Changed plugins.security.authcz.admin_dn from a plain string to a YAML list format (["CN=kirk,OU=client,O=client,L=test,C=de"])
  • The previous format caused Settings.getAsList() to split the DN on commas, resulting in 5 invalid entries instead of 1 valid DN

CI cleanup:

  • Removed macos-latest from the build matrix in .github/workflows/build.yml
  • Added bin/main/ and bin/test/ (build output directories) to .gitignore

Testing

  • All BasicReplicationIT tests pass with security enabled (-Psecurity=true)
  • Verified clusters start successfully with cert auth health checks

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@cwperks cwperks force-pushed the cert-auth-health-check branch 2 times, most recently from fb6738a to 19aea57 Compare May 1, 2026 20:46
@cwperks cwperks changed the title Use admin certificate auth for cluster health checks when security is enabled Bump to 3.7.0-SNAPSHOT and use admin certificate auth for cluster health checks May 1, 2026
@cwperks cwperks force-pushed the cert-auth-health-check branch 2 times, most recently from d8d048b to dac3f20 Compare May 1, 2026 20:51
@cwperks cwperks marked this pull request as draft May 1, 2026 21:07
Signed-off-by: Craig Perkins <cwperx@amazon.com>
@cwperks cwperks force-pushed the cert-auth-health-check branch 3 times, most recently from ba6c280 to 4318cdc Compare May 1, 2026 23:57
@cwperks cwperks changed the title Bump to 3.7.0-SNAPSHOT and use admin certificate auth for cluster health checks Bump to 3.7.0-SNAPSHOT, use admin cert auth for health checks, regenerate test certs, fix admin_dn parsing May 1, 2026
@cwperks cwperks force-pushed the cert-auth-health-check branch 3 times, most recently from daff0c5 to d3d08a7 Compare May 2, 2026 02:04
@cwperks cwperks changed the title Bump to 3.7.0-SNAPSHOT, use admin cert auth for health checks, regenerate test certs, fix admin_dn parsing Bump to 3.7.0-SNAPSHOT, fix security plugin compatibility, use admin cert auth for health checks May 2, 2026
@cwperks cwperks marked this pull request as ready for review May 2, 2026 02:06
@cwperks cwperks force-pushed the cert-auth-health-check branch 4 times, most recently from 4549666 to 3cc7039 Compare May 2, 2026 03:08
…rate test certs with IPv6 SAN, fix admin_dn parsing, implement TransportIndicesResolvingAction for CCR metadata action, remove macOS CI

Signed-off-by: Craig Perkins <cwperx@amazon.com>
@cwperks cwperks force-pushed the cert-auth-health-check branch from 3cc7039 to 02d8961 Compare May 2, 2026 03:09
@cwperks
Copy link
Copy Markdown
Member Author

cwperks commented May 2, 2026

@ankitkala @krishna-ggk @soosinha @gbbafna @monusingh-1 FYI this will fix the checks. Please review when you have some time.

While the admin cert change is tangential to fixing the security checks, I would like to introduce it now in the context of cwperks/security#92 which I would plan to add checks for and it would be needed to make API calls in light of security index initialization on a standby cluster.

Basically in this PR it makes cluster health checks use admin cert rather then basic auth creds which is a harmless change. This also alters the certs checked into this repo to add the IPv6 loopback address as a SAN and added a document to add the steps for generating these certs.

@ankitkala ankitkala merged commit 5dbfd26 into opensearch-project:main May 6, 2026
15 checks passed
mohit10011999 pushed a commit to mohit10011999/cross-cluster-replication that referenced this pull request May 9, 2026
…cert auth for health checks (opensearch-project#1667)

* Bump to 3.7.0-SNAPSHOT and remove macOS CI checks

Signed-off-by: Craig Perkins <cwperx@amazon.com>

* Bump to 3.7.0-SNAPSHOT, use admin cert auth for health checks, regenerate test certs with IPv6 SAN, fix admin_dn parsing, implement TransportIndicesResolvingAction for CCR metadata action, remove macOS CI

Signed-off-by: Craig Perkins <cwperx@amazon.com>

---------

Signed-off-by: Craig Perkins <cwperx@amazon.com>
Signed-off-by: Mohit Kumar <mohitamg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants