Translate Instance Store NVME EKS metrics by agarakan · Pull Request #1943 · aws/amazon-cloudwatch-agent

agarakan · 2025-11-15T03:05:36Z

Description of the issue

Scrape and translate NVME metrics for Instance Store on EKS. This leverages the awscontainerinsightsreceiver from the otel contrib package

Description of changes

Leverage the updated awscontainerinsightsreceiver to scrape Instance Store metrics from EKS
Add metric name mapping to conform to cloudwatch agent metric naming convention
Define dimension set
Export metrics in EMF Log format.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Unit testing

make fmt
make fmt-sh
make lint
make test

Validated EMF Log generated from manual test a test EKS cluster vending Instance Store NVME metrics.

{
    "AutoScalingGroupName": "",
    "CloudWatchMetrics": [
        {
            "Namespace": "ContainerInsights",
            "Dimensions": [
                [
                    "ClusterName"
                ],
                [
                    "ClusterName",
                    "InstanceId",
                    "NodeName"
                ],
                [
                    "ClusterName",
                    "InstanceId",
                    "NodeName",
                    "VolumeId"
                ]
            ],
            "Metrics": [
                {
                    "Name": "node_diskio_instance_store_total_write_time",
                    "Unit": "Second",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_ec2_instance_performance_exceeded_iops",
                    "Unit": "Second",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_total_read_ops",
                    "Unit": "Count",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_total_write_ops",
                    "Unit": "Count",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_total_read_time",
                    "Unit": "Second",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_total_read_bytes",
                    "Unit": "Bytes",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_volume_queue_length",
                    "Unit": "Count",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_total_write_bytes",
                    "Unit": "Bytes",
                    "StorageResolution": 60
                },
                {
                    "Name": "node_diskio_instance_store_ec2_instance_performance_exceeded_tp",
                    "Unit": "Second",
                    "StorageResolution": 60
                }
            ]
        }
    ],
    "ClusterName": "instance-store-test",
    "InstanceId": "i-05a21e33f03fe16b5",
    "InstanceType": "i3en.24xlarge",
    "NodeName": "ip-192-168-31-5.us-west-2.compute.internal",
    "PlatformType": "AWS::EKS",
    "Timestamp": "1763321337000",
    "Type": "NodeInstanceStore",
    "Version": "0",
    "VolumeId": "/dev/nvme4n1",
    "http.scheme": "http",
    "instance_id": "ip-192-168-31-5.us-west-2.compute.internal",
    "k8s.namespace.name": "kube-system",
    "kubernetes": {
        "host": "ip-192-168-31-5.us-west-2.compute.internal"
    },
    "net.host.name": "instance-store-csi-metrics.kube-system.svc",
    "net.host.port": "8101",
    "server.address": "instance-store-csi-metrics.kube-system.svc",
    "server.port": "8101",
    "service.instance.id": "instance-store-csi-metrics.kube-system.svc:8101",
    "service.name": "containerInsightsNVMeLISScraper",
    "url.scheme": "http",
    "volume_id": "/dev/nvme4n1",
    "node_diskio_instance_store_ec2_instance_performance_exceeded_iops": 0,
    "node_diskio_instance_store_ec2_instance_performance_exceeded_tp": 0,
    "node_diskio_instance_store_total_read_bytes": 73728,
    "node_diskio_instance_store_total_read_ops": 32,
    "node_diskio_instance_store_total_read_time": 0.0008310000000015805,
    "node_diskio_instance_store_total_write_bytes": 0,
    "node_diskio_instance_store_total_write_ops": 0,
    "node_diskio_instance_store_total_write_time": 0,
    "node_diskio_instance_store_volume_queue_length": 1
}

Requirements

Before commiting your code, please do the following steps.

Run make fmt and make fmt-sh
Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

agarakan · 2025-11-18T20:21:51Z

internal/containerinsightscommon/const.go

+	NvmeInstanceStoreWriteTime           = "diskio_instance_store_total_write_time"
+	NvmeInstanceStoreExceededEC2IOPSTime = "diskio_instance_store_ec2_instance_performance_exceeded_iops"
+	NvmeInstanceStoreExceededEC2TPTime   = "diskio_instance_store_ec2_instance_performance_exceeded_tp"
+	NvmeInstanceStoreVolumeQueueLength   = "diskio_instance_store_volume_queue_length"


Please note there are 2 EBS NVMe metrics that do not have an equivalent metric for LIS NVMe metrics. This is called out in the design, and communicated by the team owning the LIS CSI that these volume performance metrics are not relevant for LIS CSI. Additionally we have existing unit tests confirming those 2 metrics are not supported.

// relevant EBS iops/tp metrics supported
NvmeExceededIOPSTime = "diskio_ebs_volume_performance_exceeded_iops"
NvmeExceededTPTime = "diskio_ebs_volume_performance_exceeded_tp"
NvmeExceededEC2IOPSTime = "diskio_ebs_ec2_instance_performance_exceeded_iops"
NvmeExceededEC2TPTime = "diskio_ebs_ec2_instance_performance_exceeded_tp"

//relevant LIS iops/tp metrics supported
NvmeInstanceStoreExceededEC2IOPSTime = "diskio_instance_store_ec2_instance_performance_exceeded_iops"
NvmeInstanceStoreExceededEC2TPTime = "diskio_instance_store_ec2_instance_performance_exceeded_tp"

translator/tocwconfig/sampleConfig/emf_and_kubernetes_config.yaml

mitali-salvi · 2025-11-18T20:21:38Z

go.mod

-	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.0.0-20251103165826-c3d3976f1fa0
-	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsemfexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsemfexporter v0.0.0-20251103165826-c3d3976f1fa0
-	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awsxrayexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awsxrayexporter v0.0.0-20251103165826-c3d3976f1fa0
+	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter => github.com/amazon-contributing/opentelemetry-collector-contrib/exporter/awscloudwatchlogsexporter v0.0.0-20251118152320-cd34549ef7ed


is this from this one ? otel contrib PR - amazon-contributing/opentelemetry-collector-contrib#382

Yes correct

mitali-salvi · 2025-11-18T20:24:21Z

translator/translate/otel/processor/metricstransformprocessor/translator.go

+				transformRules = append(transformRules, map[string]interface{}{
+					"include":  oldNvmeInstanceStoreMetric,
+					"action":   "update",
+					"new_name": containerinsightscommon.MetricName(containerinsightscommon.TypeNode, newNvmeInstanceStoreMetric),


should the type be TypeNodeInstanceStore here instead of Node ?

No, the prefix is actually "node". The TypeNodeInstanceStoreis used as the MetricType label on line 141

dricross · 2025-11-18T21:45:26Z

There are some failing tests, but I don't think they are related to these changes (xray and selinux/emf). Probably just flaky. Give them a retry. We should look at those though (outside this PR of course). Approving

agarakan · 2025-11-18T23:47:37Z

There are some failing tests, but I don't think they are related to these changes (xray and selinux/emf). Probably just flaky. Give them a retry. We should look at those though (outside this PR of course). Approving

Tests succeeded on rerun after github internal issues were resolved

translator/tocwconfig/sampleConfig/emf_and_kubernetes_config.yaml

jefchien · 2025-11-19T00:06:39Z

translator/translate/otel/exporter/awsemf/kubernetes.go

+					"node_diskio_instance_store_total_read_ops",
+					"node_diskio_instance_store_total_write_ops",
+					"node_diskio_instance_store_total_read_bytes",
+					"node_diskio_instance_store_total_write_bytes",
+					"node_diskio_instance_store_total_read_time",
+					"node_diskio_instance_store_total_write_time",
+					"node_diskio_instance_store_ec2_instance_performance_exceeded_iops",
+					"node_diskio_instance_store_ec2_instance_performance_exceeded_tp",
+					"node_diskio_instance_store_volume_queue_length",


nit: Is there a reason we aren't using the consts defined in containerinsightscommon?

Yeah good question, I decided to follow the existing convention in this file, as all the other kubernetes metrics have the same format. I'm looking to update the entire file to depend on constants in a future cr as a refactor

jefchien · 2025-11-19T00:09:17Z

translator/translate/otel/processor/metricstransformprocessor/translator.go

 	"aws_ebs_csi_volume_queue_length":             containerinsightscommon.NvmeVolumeQueueLength,
 }

+var renameMapForNvmeInstanceStore = map[string]string{


Oh interesting. Had no idea the mapping was being done in the processor and not the prometheus receiver, but this makes sense.

jefchien · 2025-11-19T00:17:31Z

internal/containerinsightscommon/util_test.go

+			if result != tc.expectedName {
+				t.Errorf("MetricName(%q, %q) = %q, want %q", tc.metricType, tc.metricName, result, tc.expectedName)
+			}


nit: For consistency, consider using github.com/stretchr/testify/assert.

thanks, will update in a follow up

jefchien · 2025-11-19T00:18:10Z

internal/containerinsightscommon/util_test.go

+	}
+}
+
+func TestMetricNameWithRealConstants(t *testing.T) {


nit: What is this testing that TestMetricName isn't?

its testing the method called MetricName to ensure it actually performs prefix injection appropriately.

agarakan added 8 commits November 15, 2025 02:53

Translate Instance Store NVME EKS metrics

6049445

lint

3bd7f02

Add new metricselectors to config test expectations

8d53415

remove hardcoded metric strings

69270c1

add unit testing around metricName logic AKA metric renaming

4a0be8e

revert to metric strings. will revisit later

686df33

update emf exporter unit test

a0571d0

Update go.mod with latest otel collector contrib

efc54d7

agarakan marked this pull request as ready for review November 18, 2025 19:18

agarakan requested a review from a team as a code owner November 18, 2025 19:18

agarakan added the ready for testing Indicates this PR is ready for integration tests to run label Nov 18, 2025

dricross previously approved these changes Nov 18, 2025

View reviewed changes

agarakan added 2 commits November 18, 2025 19:45

Bump otel dependencies

24938e8

Lint

cc88bc1

agarakan dismissed dricross’s stale review via cc88bc1 November 18, 2025 19:56

agarakan commented Nov 18, 2025

View reviewed changes

mitali-salvi reviewed Nov 18, 2025

View reviewed changes

mitali-salvi approved these changes Nov 18, 2025

View reviewed changes

musa-asad approved these changes Nov 18, 2025

View reviewed changes

TravisStark approved these changes Nov 18, 2025

View reviewed changes

dricross approved these changes Nov 18, 2025

View reviewed changes

Paramadon approved these changes Nov 18, 2025

View reviewed changes

jefchien approved these changes Nov 19, 2025

View reviewed changes

agarakan merged commit 7bf323e into main Nov 19, 2025
491 of 497 checks passed

agarakan deleted the lis-csi-nvme-metric-support branch November 19, 2025 00:48

agarakan mentioned this pull request Nov 20, 2025

Introduce integ tests for NVME Instance Store metrics for Container Insights aws/amazon-cloudwatch-agent-test#621

Merged

sky333999 mentioned this pull request Dec 2, 2025

Add Nvme scrape configs for LIS CSI amazon-contributing/opentelemetry-collector-contrib#382

Merged

Comments

Conversation

agarakan commented Nov 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of the issue

Description of changes

License

Tests

Requirements

Integration Tests

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dricross commented Nov 18, 2025

Uh oh!

agarakan commented Nov 18, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

agarakan commented Nov 15, 2025 •

edited

Loading