[BUG] OpenSearch container startup extremely slow on VMs due to low entropy

[todest]

Describe the bug

When running OpenSearch in a containerized environment on virtual machines, we have observed extremely slow startup times — in some cases taking over 1–2 hours to complete initialization. Upon investigation, the root cause appears to be low system entropy, with /proc/sys/kernel/random/entropy_avail consistently showing values below 200, and sometimes even in the double digits.

use -Djava.security.egd=file:/dev/urandom has no effect

Related component

Other

To Reproduce

1、run cat /proc/sys/kernel/random/entropy_avail （<200, in aliyun ecs)
2、run docker run -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node" -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=<password>" opensearchproject/opensearch:3
3、logs stuck in

WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens

4、run haveged -F -d 32 -w 1024 --verbose=1
5、problem solved

Expected behavior

Is there any way to solve this problem without relying on external programs (such as haveged)?

[HenryTheSir]

Hi,

we observed the same slow behaviour starting with 3.1. in 3.0 the startup was faster.

best regards

[cwperks]

See #18729 (comment)

This will be fixed in 3.2: #18758

[todest]

Thanks, but I want to know when 3.2 will be released? Will the docker image be released along with it?

[HenryTheSir]

Hi

I tested with current 3.2 Docker Image and the error still exists that the startup is very slow (1-2 hours).

best regards

[cwperks]

Hi

I tested with current 3.2 Docker Image and the error still exists that the startup is very slow (1-2 hours).

best regards

@HenryTheSir is there any more info you can provide about your docker configuration?

[YeonghyeonKO]

@HenryTheSir @cwperks
Not just by deploying with docker, I ran into the same issue when deploying a cluster with OpenSearchCluster CR (based on opensearch-k8s-operator 2.8.0).

Some of nodes(no matter with their role) never go further than just being stuck like the below logs:

Defaulted container "opensearch" out of: opensearch, init (init), init-sysctl (init)
Enabling OpenSearch Security Plugin
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
OpenSearch 2.12.0 onwards, the OpenSearch Security Plugin a change that requires an initial password for 'admin' user.
Please define an environment variable 'OPENSEARCH_INITIAL_ADMIN_PASSWORD' with a strong password string.
If a password is not provided, the setup will quit.
 For more details, please visit: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/docker/
### OpenSearch Security Demo Installer
### ** Warning: Do not use on production or public reachable systems **
OpenSearch install type: rpm/deb on Linux 4.18.0-553.34.1.el8_10.x86_64 amd64
OpenSearch config dir: /usr/share/opensearch/config/
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin/
OpenSearch plugins dir: /usr/share/opensearch/plugins/
OpenSearch lib dir: /usr/share/opensearch/lib/
Detected OpenSearch Version: 3.2.0
Detected OpenSearch Security Version: 3.2.0.0
/usr/share/opensearch/config/opensearch.yml seems to be already configured for Security. Quit.
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: Using incubator modules: jdk.incubator.vector
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
WARNING: A terminally deprecated method in sun.misc.Unsafe has been called
WARNING: sun.misc.Unsafe::objectFieldOffset has been called by net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: Please consider reporting this to the maintainers of class net.bytebuddy.dynamic.loading.ClassInjector$UsingUnsafe$Dispatcher$CreationAction
WARNING: sun.misc.Unsafe::objectFieldOffset will be removed in a future release

https://forum.opensearch.org/t/docker-image-3-1-1-doesnt-seem-to-work/24875/30

[peterzhuamazon]

I suspect this is due to PA cli, since RCA has been deprecated but we never see the issue in our test env.
https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/README.md#disable-performance-analyzer-agent-cli-and-related-configurations

You can try to disable PA cli through a env var such as -e "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true".

Let me know if that helps @HenryTheSir @YeonghyeonKO. Thanks.

[YeonghyeonKO]

Hi @peterzhuamazon ! I will try to disable PA cli and write the result soon. Thank you

[YeonghyeonKO]

@peterzhuamazon @HenryTheSir ,
I found that two of my worker nodes in k8s shows a dramatic drop with entropy value after some pods being hosted on those nodes are being stuck. (4022 -> 24)

$ watch cat /proc/sys/kernel/random/entropy_avail
4022   # (before) 
...
24     # (after) tried to host a OpenSearch cluster but failed 

I believe somewhat of OpenSearch components make the entropy exhausted and prohibit generating random number (ID, SSL object). AFAIK, OpenSearch uses /dev/random to generate random numbers for encrypting objects such as IDs and SSL traffic in order to enhance security. (https://github.com/opensearch-project/OpenSearch/blob/3.2.0/server/src/main/java/org/opensearch/common/Randomness.java#L138)

[root@bt001 ~]# ssh admin@opensearchw001 "cat /proc/sys/kernel/random/entropy_avail"
3941
[root@bt001 ~]# ssh admin@opensearchw002 "cat /proc/sys/kernel/random/entropy_avail" 
3945
[root@bt001 ~]# ssh admin@opensearchw003 "cat /proc/sys/kernel/random/entropy_avail" 
3922
[root@bt001 ~]# ssh admin@opensearchw004 "cat /proc/sys/kernel/random/entropy_avail" 
3619
[root@bt001 ~]# ssh admin@opensearchw005 "cat /proc/sys/kernel/random/entropy_avail" 
3692
[root@bt001 ~]# ssh admin@opensearchw006 "cat /proc/sys/kernel/random/entropy_avail" 
3961
[root@bt001 ~]# ssh admin@opensearchw007 "cat /proc/sys/kernel/random/entropy_avail" 
3816
[root@bt001 ~]# ssh admin@opensearchw008 "cat /proc/sys/kernel/random/entropy_avail" 
8
[root@bt001 ~]# ssh admin@opensearchw009 "cat /proc/sys/kernel/random/entropy_avail" 
56
[root@bt001 ~]# ssh admin@opensearchw010 "cat /proc/sys/kernel/random/entropy_avail"  
3803

$ kubectl get po -o wide
NAME                                                          READY   STATUS    RESTARTS          AGE   IP               NODE             NOMINATED NODE   READINESS GATES
dev-opensearch-bootstrap-0                   1/1     Running   8 (11h ago)       11h   10.251.143.225   opensearchw002   <none>           <none>
dev-opensearch-coordinating-0                0/1     Running   127 (7m37s ago)   11h   10.251.175.252   opensearchw008   <none>           <none>
dev-opensearch-dashboards-f95547b8-9n7tk     0/1     Running   125 (3m20s ago)   11h   10.251.175.220   opensearchw008   <none>           <none>
dev-opensearch-data-0                        0/1     Running   125 (47s ago)     11h   10.251.175.234   opensearchw008   <none>           <none>
dev-opensearch-ingest-0                      0/1     Running   10 (11h ago)      11h   10.251.143.194   opensearchw002   <none>           <none>
dev-opensearch-master-0                      0/1     Running   8 (11h ago)       11h   10.251.117.170   opensearchw005   <none>           <none>
dev-opensearch-ml-0                          0/1     Running   125 (2m17s ago)   11h   10.251.175.195   opensearchw008   <none>           <none>
dev-opensearch-securityconfig-update-shntm   1/1     Running   0                 11h   10.251.175.222   opensearchw008   <none>           <none>

[YeonghyeonKO]

I suspect this is due to PA cli, since RCA has been deprecated but we never see the issue in our test env. https://github.com/opensearch-project/opensearch-build/blob/main/docker/release/README.md#disable-performance-analyzer-agent-cli-and-related-configurations

You can try to disable PA cli through a env var such as -e "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true".

Let me know if that helps @HenryTheSir @YeonghyeonKO. Thanks.

Indeed, PerformanceAnalayzer consumes linux entropy when creating SSLContext in PerformanceAnalyzerResourceProvider.java

[YeonghyeonKO]

@peterzhuamazon , I tried to disable Performance Analyzer Agent CLI by injecting DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true env to additionalConfig in CR (OpenSearchCluster) but it didn't work.

cluster:
  name: "test-opensearch-cluster-v3-2-0" 
  bootstrap:
    additionalConfig: 
      DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI: "true"
  ...
  nodePools:
  - additionalConfig:
      DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI: "true"
  ...