Test OpenSearch 3.4.0

[siegfriedweber]

Test result

OpenSearch 3.4.0 can be used in SDP 26.3.

Findings

• Gradle was upgraded from version 8.14.3 to 9.2.0. Therefore, the CycloneDX Gradle Plugin must be upgraded to version 3.x which is compatible with Gradle 9.x.

  Solution:

  • Upgrade the CycloneDX Gradle Plugin to version 3.1.0 and adapt its configuration.
  • Upgrade com.networknt:json-schema-validator to version 1.5.9.
  • Upgrade jackson and jackson-databind to version 2.20.1.
  • Upgrade jackson-annotations to version 2.20.

• The internal communication seems to use pod IPs:

  java.security.cert.CertificateException: No subject alternative names matching IP address 10.42.0.72 found
  

  There is already an issue to add the pod IP to the certificate: Add missing Pod IPs to created certificates secret-operator#620
  A workaround is to set transport.ssl.enforce_hostname_verification: false. The hostname verification was also enabled in OpenSearch 3.1.0 but there it worked for whatever reason.
  This problem also exists in OpenSearch 3.3.0. OpenSearch 3.2.0 works.

  Related links:

  • https://www.github.com/opensearch-project/OpenSearch/pull/19458
  • https://www.github.com/opensearch-project/security/pull/5667
  • https://forum.opensearch.org/t/certificate-unknown-in-opensearch-3-3/27308
  • https://forum.opensearch.org/t/exception-during-establishing-a-ssl-connection-javax-net-ssl-sslhandshakeexception-after-upgrading-to-opensearch-3-3/27381

  Solution:

  • Set network.publish_host, transport.publish_host and http.publish_host to the fully-qualified domain name, i.e. <pod-name>.<headless-service-name>.<namespace>.svc.<cluster-domain.
  • Use the FQDN for discovery.seed_hosts.

• The TLS negotiation between OpenSearch and MinIO fails.
  Test data is created in the S3 bucket, but OpenSearch cannot delete it anymore. The stack trace contains "TLS negotiation failed".
  The backup-restore integration test passes if TLS is disabled in MinIO.

  The S3CrtClient is now used instead of Netty (see https://www.github.com/opensearch-project/OpenSearch/issues/18535).

  Solution:
  If the S3CrtClient cannot be configured to connect to the S3 instance, use Netty instead when creating the snapshot repository:

  client.snapshot.create_repository(
    repository="snapshot_repository",
    body={
      "type": "s3",
      "settings": {
        "bucket": "opensearch-data",
        "s3_async_client_type": "netty"
      }
    }
  )

  The option s3_async_client_type is unfortunately not documented.

• The script securityadmin.sh is not executable anymore. In the official image, it is.

  The scripts are executable in the source code of the security plugin. The gradle build produces build/distributions/opensearch-security-<version>.zip. In version 3.1.0, the scripts in this zip file are executable, in 3.4.0, they are not for whatever reason.

  Solution:
  Just make the scripts executable again in the Dockerfile. The offical image does the same (see https://github.com/stackabletech/opensearch-build/blob/3.4.0/docker/release/dockerfiles/opensearch.al2.dockerfile#L47).

• OpenSearch Dashboards just works.