[BUG] OpenSearch 3.1.0 freezes when running on AlmaLinux 8

[pawelw1]

Describe the bug

This issue was originally reported in the OpenSearch forum, and all investigation/troubleshooting were performed there.

https://forum.opensearch.org/t/docker-image-3-1-1-doesnt-seem-to-work/24875

OpenSearch 3.1.0 freezes when running on AlmaLinux 8 after these OpenSearch log lines

WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to 

There are no errors in the Opensearch logs or journalctl that would indicate the root cause.

The issue is visible with Docker deployment and with a minimal binary version.

OpenSearch 3.0.0 starts successfully.

Related component

Other

To Reproduce

Deploy LinuxAlma 8 VM and deploy OpenSearch 3.1.0 as a Docker container or binary.

Expected behavior

OpenSearch 3.1.0 should fully start as 3.0.0 does.

Additional Details

Host/Environment (please complete the following information):

• OS: VMware VM AlamaLinux 8

Additional context

It was discovered that OpenSearch 3.1.0 starts successfully on LinuxAlma 9 and 10. After comparing all the versions, it was found that LinuxAlma 8 runs on kernel 4.18.0, 9 - 5.x and 10 - 6.x

Installing kernel 5.x on AlmaLinux 8 allowed the OpenSearch 3.1.0 to start successfully

[root@localhost pablo]# curl --insecure -u admin:Eliatra123 https://localhost:9200
{
  "name" : "opensearch-node1",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "BTKf7poWR3OosXwDZQ07og",
  "version" : {
    "distribution" : "opensearch",
    "number" : "3.1.0",
    "build_type" : "tar",
    "build_hash" : "8ff7c6ee924a49f0f59f80a6e1c73073c8904214",
    "build_date" : "2025-06-21T08:05:43.345081313Z",
    "build_snapshot" : false,
    "lucene_version" : "10.2.1",
    "minimum_wire_compatibility_version" : "2.19.0",
    "minimum_index_compatibility_version" : "2.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[root@localhost pablo]# cat /etc/*release
AlmaLinux release 8.10 (Cerulean Leopard)
AlmaLinux release 8.10 (Cerulean Leopard)
NAME="AlmaLinux"
VERSION="8.10 (Cerulean Leopard)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.10 (Cerulean Leopard)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.10"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"
SUPPORT_END=2029-06-01
AlmaLinux release 8.10 (Cerulean Leopard)
AlmaLinux release 8.10 (Cerulean Leopard)

[root@localhost pablo]# uname -r
5.4.295-1.el8.elrepo.x86_64

[peterzhuamazon]

I can only speak for the infra side when creating the artifacts.
We are using the latest AL2023 as host and running almalinux8 as docker for testing (RPM).
They already have 5.x+ kernel on the host so we never see this issue on our side.
https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2-kernel.html

Based on this chart it seems indeed Almalinux8 is shipping 4.x kernel by default:
https://wiki.almalinux.org/release-notes/

cc @andrross @reta from core side to understand what has changed in core that may have caused this.

Thanks.

[andrross]

Very strange. I just spun up an AlmaLinux instance with kernel 4.18.0 and verified the hanging behavior is reproducible with the min distribution tarball. The ./bin/opensearch startup script gets to the line where it invokes the java binary and prints the warning about modules and then hangs (the same behavior as reported here).

The JDK seems to be functional:

[ec2-user@ip-172-31-40-217 opensearch-3.1.0]$ ./jdk/bin/java -version
openjdk version "21.0.7" 2025-04-15 LTS
OpenJDK Runtime Environment Temurin-21.0.7+6 (build 21.0.7+6-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.7+6 (build 21.0.7+6-LTS, mixed mode, sharing)

[andrross]

I took a stack trace of the hung process, which shows the bootstrap code stuck at this point:

"main" #1 [4157] prio=5 os_prio=0 cpu=790.01ms elapsed=136.65s tid=0x00007fd2c0035a40 nid=4157 runnable  [0x00007fd2c6927000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(java.base@21.0.7/Native Method)
	at java.io.FileInputStream.read(java.base@21.0.7/FileInputStream.java:287)
	at java.io.FilterInputStream.read(java.base@21.0.7/FilterInputStream.java:119)
	at sun.security.provider.NativePRNG$RandomIO.readFully(java.base@21.0.7/NativePRNG.java:432)
	at sun.security.provider.NativePRNG$RandomIO.getMixRandom(java.base@21.0.7/NativePRNG.java:412)
	- locked <0x00000000fe0e01b0> (a java.lang.Object)
	at sun.security.provider.NativePRNG$RandomIO.implNextBytes(java.base@21.0.7/NativePRNG.java:544)
	at sun.security.provider.NativePRNG$Blocking.engineNextBytes(java.base@21.0.7/NativePRNG.java:274)
	at java.security.SecureRandom.nextBytes(java.base@21.0.7/SecureRandom.java:768)
	at java.security.SecureRandom.next(java.base@21.0.7/SecureRandom.java:824)
	at java.util.Random.nextInt(java.base@21.0.7/Random.java:558)
	at org.opensearch.common.settings.KeyStoreWrapper.addBootstrapSeed(KeyStoreWrapper.java:225)
	at org.opensearch.common.settings.KeyStoreWrapper.create(KeyStoreWrapper.java:214)
	at org.opensearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:282)
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:375)
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168)
	at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159)
	at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110)
	at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
	at org.opensearch.cli.Command.main(Command.java:101)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125)
	at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91)

PR #17907 did introduce some changes to similar code (though nothing directly I see in this stack trace). @cwperks any ideas here why this would hang indefinitely on Linux kernel 4.18.0 but not newer versions?

[cwperks]

Potentially related: quarkusio/quarkus#26736

CC: @beanuwave

[andrross]

I suspect this is the change that caused this:

OpenSearch/server/src/main/java/org/opensearch/common/Randomness.java

Line 172 in 32cd2e6

return SecureRandom.getInstanceStrong();

Previously in non-FIPS mode Randomness#createSecure would return new SecureRandom(). PR #17907 changed it to return SecureRandom.getInstanceStrong()

[cwperks]

Perhaps we can make the strength level of it configurable with a setting?

Edit: Found a relevant test from a few years back: 4a8cd45

[reta]

Perhaps we can make the strength level of it configurable with a setting?

I am wondering if changing entropy source would help? -Djava.security.egd=file:/dev/./urandom

[beanuwave]

AFAIU -Djava.security.egd has no effect on the algorithm selection for SecureRandom.getInstanceStrong(), since that is driven exclusively by securerandom.strongAlgorithms. Should we return to new SecureRandom() instead? It's less explicit about which algorithm gets picked, but seems to be more supportive of older environments per default.

[andrross]

Should we return to new SecureRandom() instead? It's less explicit about which algorithm gets picked, but seems to be more supportive of older environments per default.

@beanuwave Unless there's a really compelling reason to use SecureRandom.getInstanceStrong() then we should probably go back to new SecureRandom() since this is impacting some users.