[BUG] (v3.2.0) `_cat/nodes` API shows negative values in cpu stats

[YeonghyeonKO]

Describe the bug

I ran into the below issue in 3.2.0 version that cpu usages could be negative and the same value (-1)

$ curl "https://dev-opensearch-test-250821-os.osp01.abc.com/_cat/nodes?v" -ku admin:test

ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles      cluster_manager name
10.251.175.203           38          60  -1    0.44    0.41     0.49 m         cluster_manager *               dev-opensearch-test-250821-master-1
10.251.117.144           20          60  -1    0.58    0.54     0.60 m         cluster_manager -               dev-opensearch-test-250821-master-0
10.251.225.190           67          90  -1    1.79    1.14     1.03 di        data,ingest     -               dev-opensearch-test-250821-data-0
10.251.143.196           24          60  -1    0.40    0.68     0.61 m         cluster_manager -               dev-opensearch-test-250821-master-2

Related component

Clients

To Reproduce

1. deploy opensearch cluster using 3.2.0 version
2. After your cluster runs successfully, curl "https://dev-opensearch-test-250821-os.osp01.abc.com/_cat/nodes?v" -ku admin:test
3. It also happens wrongly when you try _cluster/stats API

Expected behavior

.

Additional Details

Plugins
I just added the prometheus-exporter plugin by building new Docker image

FROM harbor.abc.com/library/opensearchproject/opensearch:3.2.0

COPY prometheus-exporter-3.2.0.0.zip /tmp/
RUN ./bin/opensearch-plugin install file:/tmp/prometheus-exporter-3.2.0.0.zip

Host/Environment (please complete the following information):

• OS: All hosts are worker nodes of kubernetes v1.31.4 and based on RHEL 8.10 (4.18.0 kernel)
• Version: OpenSearch 3.2.0

Additional context

• Air-gapped environment but I believe it doesn't matter with this issue.

[nateynateynate]

To help triage, I followed the steps but without installing the prometheus exporter plugin, and I was unable to reproduce.

• OS: Mac (Sequoia 15.6)
• Version: 3.2.0 Docker Image

[YeonghyeonKO]

Hi @nateynateynate ,
I will check if it is able to reproduce this issue with the new cluster without prometheus-exporter. Thanks!

[YeonghyeonKO]

@nateynateynate ,
The below experiment is implemented using docker compose (macOS Sequoia 15.6.1) and there seem to have no issue as you mentioned. (even deploying with prometheus-exporter)

I will investigate further in our Kubernetes environment (RHEL 8.10, 4.18.0 kernel) to see if the issue is reproducible only in that environment.

---

3.2.0 WITHOUT prometheus-exporter

➜  OPENSEARCH_VERSION=3.2.0 docker compose up --remove-orphans -d                 
[+] Running 14/14
 ✔ Network osb_opensearch-net                Created     0.0s 
...
 ✔ Container opensearch-dashboards       Started     2.7s 

➜  curl "https://localhost:9200/_cat/nodes?v" -ku admin:userStrongPassword123!            
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles      cluster_manager name
172.20.0.3           80         100  15    0.57    3.11     4.03 m         cluster_manager *               opensearch-clustermanager-2
172.20.0.7           53          84  13    0.57    3.11     4.03 di        data,ingest     -               opensearch-data-2
172.20.0.6           78         100  14    0.57    3.11     4.03 m         cluster_manager -               opensearch-clustermanager-3
172.20.0.5           51         100  15    0.57    3.11     4.03 m         cluster_manager -               opensearch-clustermanager-1
172.20.0.8           72         100  15    0.57    3.11     4.03 -         -               -               opensearch-coordinating-1
172.20.0.4           36          85  13    0.57    3.11     4.03 di        data,ingest     -               opensearch-data-1

➜  curl "https://localhost:9200/_cat/plugins" -ku admin:userStrongPassword123! | grep prom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11388  100 11388    0     0  31082      0 --:--:-- --:--:-- --:--:-- 31114

➜  curl "https://localhost:9200/_prometheus/metrics" -ku admin:userStrongPassword123!     
{"error":"no handler found for uri [/_prometheus/metrics] and method [GET]"}% 

3.2.0 WITH prometheus-exporter

➜  vi docker-compose.yaml                                         
➜  OPENSEARCH_VERSION=3.2.0-plugins docker compose up --remove-orphans -d    
[+] Running 14/14
 ✔ Network osb_opensearch-net                Created                                                                      0.1s 
...
 ✔ Container opensearch-clustermanager-2     Started                                                                      2.8s 

➜  curl "https://localhost:9200/_cat/nodes?v" -ku admin:userStrongPassword123!
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles      cluster_manager name
172.19.0.6           39          84  10   11.54    7.56     5.13 di        data,ingest     -               opensearch-data-2
172.19.0.8           65         100  10   11.54    7.56     5.13 m         cluster_manager *               opensearch-clustermanager-3
172.19.0.3           41         100  11   11.54    7.56     5.13 m         cluster_manager -               opensearch-clustermanager-2
172.19.0.5           70          84  10   11.54    7.56     5.13 di        data,ingest     -               opensearch-data-1
172.19.0.2           49         100  10   11.54    7.56     5.13 -         -               -               opensearch-coordinating-1
172.19.0.4           71         100  10   11.54    7.56     5.13 m         cluster_manager -               opensearch-clustermanager-1

➜  curl "https://localhost:9200/_cat/plugins" -ku admin:userStrongPassword123! | grep prom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11826  100 11826    0     0   229k      0 --:--:-- --:--:-- --:--:--  230k
opensearch-data-2           prometheus-exporter                  3.2.0.0
opensearch-clustermanager-3 prometheus-exporter                  3.2.0.0
opensearch-clustermanager-2 prometheus-exporter                  3.2.0.0
opensearch-data-1           prometheus-exporter                  3.2.0.0
opensearch-coordinating-1   prometheus-exporter                  3.2.0.0
opensearch-clustermanager-1 prometheus-exporter                  3.2.0.0

➜  curl "https://localhost:9200/_prometheus/metrics" -ku admin:userStrongPassword123!            
# HELP opensearch_indices_search_fetch_count Count of search fetches
# TYPE opensearch_indices_search_fetch_count gauge
opensearch_indices_search_fetch_count{cluster="opensearch-cluster-1",node="opensearch-coordinating-1",nodeid="VynhGKiwRTi4efeE1KE_SQ",} 0.0
# HELP opensearch_indices_percolate_time_seconds Time spent while percolating
# TYPE opensearch_indices_percolate_time_seconds gauge
# HELP opensearch_threadpool_tasks_number Number of tasks in thread pool
# TYPE opensearch_threadpool_tasks_number gauge
opensearch_threadpool_tasks_number{cluster="opensearch-cluster-1",node="opensearch-coordinating-1",nodeid="VynhGKiwRTi4efeE1KE_SQ",name="remote_recovery",type="queue",} 0.0
opensearch_threadpool_tasks_number{cluster="opensearch-cluster-1",node="opensearch-coordinating-1",nodeid="VynhGKiwRTi4efeE1KE_SQ",name="opensearch_workflow",type="queue",} 0.0
opensearch_threadpool_tasks_number{cluster="opensearch-cluster-1",node="opensearch-coordinating-1",nodeid="VynhGKiwRTi4efeE1KE_SQ",name="generic",type="queue",} 0.0
...

[YeonghyeonKO]

The below experiment was implemented in Ubuntu 20.04 (Focal) (CPU Usage is -1) (NOT K8s environment)

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
...

$ uname -a
Linux {hostname} 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

---

$ docker image ls | grep opensearch
opensearchproject/opensearch               3.2.0-plugins     87eb1dcb000c   4 days ago     1.45GB
opensearchproject/opensearch               3.2.0             b6dde07329e6   11 days ago    1.45GB
opensearchproject/opensearch-dashboards    3.2.0             5d4f6d03853f   11 days ago    1.69GB
opensearchproject/opensearch-operator      2.8.0             a57c19e886bf   2 months ago   73.3MB

$ docker run -d -p 9200:9200 -p 9600:9600 opensearchproject/opensearch:3.2.0 \
  -e "discovery.type=single-node" \
  -e "OPENSEARCH_INITIAL_ADMIN_PASSWORD=strongUserPassword1!"

$ curl "https://localhost:9200/_cat/nodes?v" -ku admin:strongUserPassword1!
ip         heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles                                        cluster_manager name
172.17.0.2           62          42  -1    0.03    0.13     0.10 dimr      cluster_manager,data,ingest,remote_cluster_client *               4b8ab8dace73

[YeonghyeonKO]

It seems @JakeHardy14 just has the same issue with mine : opensearch-project/opensearch-prometheus-exporter#386

[RTJoe]

I have the same thing deploying with the Kubernetes operator

https://github.com/opensearch-project/opensearch-k8s-operator

and running the command
GET _cat/nodes?v=true&s=cpu:desc

ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role node.roles           cluster_manager name
10.244.5.210           29          97  -1    1.18    1.08     1.22 dm        cluster_manager,data -               opensearch-cluster-masters-1
10.244.2.104           48         100  -1    1.17    1.00     0.96 dm        cluster_manager,data *               opensearch-cluster-masters-2
10.244.3.107           42          94  -1    1.80    1.05     0.82 dm        cluster_manager,data -               opensearch-cluster-masters-0
10.244.2.105           69          99  -1    1.17    1.00     0.96 dm        cluster_manager,data -               opensearch-cluster-masters-4
10.244.7.39            31         100  -1    0.32    0.43     0.69 dm        cluster_manager,data -               opensearch-cluster-masters-3
10.244.6.156           25          84  -1    1.52    2.22     1.53 dm        cluster_manager,data -               opensearch-cluster-masters-5

which uses these images

docker.io/opensearchproject/opensearch:3.2.0

[krisfreedain]

Catch All Triage - 1 2

[YeonghyeonKO]

@krisfreedain @andrross Hi,
Actually I didn't attend the triage meeting so would you be available to get back to this Issue if someone will take a look into this CPU metric problem?

[cinhtau]

Any update on this? It is crucial for our monitoring to get the proper cpu usage. I am using 3.3.2

Even if we use the internal nodes/stats it gives a negative -1

GET /_nodes/stats/os,process?human&pretty

This is our output

"os": {
        "timestamp": 1763470630966,
        "cpu": {
          "percent": -1,
          "load_average": {
            "1m": 1.67,
            "5m": 1.24,
            "15m": 1.36
          }
        },
        "mem": {
          "total": "125.5gb",
          "total_in_bytes": 134820294656,
          "free": "472.5mb",
          "free_in_bytes": 495480832,
          "used": "125gb",
          "used_in_bytes": 134324813824,
          "free_percent": 0,
          "used_percent": 100
        },
        "swap": {
          "total": "3.9gb",
          "total_in_bytes": 4294963200,
          "free": "853.2mb",
          "free_in_bytes": 894705664,
          "used": "3.1gb",
          "used_in_bytes": 3400257536
        },
        "cgroup": {
          "cpuacct": {
            "control_group": "/",
            "usage_nanos": 1385193963381144
          },
          "cpu": {
            "control_group": "/",
            "cfs_period_micros": 100000,
            "cfs_quota_micros": -1,
            "stat": {
              "number_of_elapsed_periods": 0,
              "number_of_times_throttled": 0,
              "time_throttled_nanos": 0
            }
          },
          "memory": {
            "control_group": "/system.slice/opensearch.service",
            "limit_in_bytes": "9223372036854771712",
            "usage_in_bytes": "128936591360"
          }
        }
      },
      "process": {
        "timestamp": 1763470630967,
        "open_file_descriptors": 12477,
        "max_file_descriptors": 1048576,
        "cpu": {
          "percent": -1,
          "total": "2.6d",
          "total_in_millis": 231113240
        },
        "mem": {
          "total_virtual": "10.4tb",
          "total_virtual_in_bytes": 11525761384448
        }
      }

[andrross]

I'm not able to reproduce this on Ubuntu 24.04.3 running the tarball or the docker image. In all cases I get non-negative value.

Here is the code in question:

OpenSearch/server/src/main/java/org/opensearch/monitor/os/OsProbe.java

Lines 239 to 241 in 631bed4

	public short getSystemCpuPercent() {
	return Probes.getLoadAndScaleToPercent(getSystemCpuLoad, osMxBean);
	}

To summarize, it uses reflection to invoke com.sun.management.OperatingSystemMXBean#getCpuLoad. It does have a couple fallback paths to return -1 in case that class or method is not available at runtime.

@dbwiddis I believe you have some experience in this area. Any idea what might be going on here?