Skip to content

Comments

Support metadata resolution for multi-cloud envs#2032

Open
movence wants to merge 11 commits intofeature-multi-cloudfrom
hsookim/multi-cloud/metadata
Open

Support metadata resolution for multi-cloud envs#2032
movence wants to merge 11 commits intofeature-multi-cloudfrom
hsookim/multi-cloud/metadata

Conversation

@movence
Copy link
Contributor

@movence movence commented Feb 18, 2026

Description of changes

Adds multi-cloud support for metric decoration and placeholder resolution. Azure (and future cloud providers)
use OTel's resourcedetectionprocessor for metadata, while EC2 continues using ec2tagger. Disk/volume tagging
is extracted into a standalone disktagger processor that supports both AWS and Azure.

Also include test fixes from #2010 and minor update to remove an unnecessary wrapper function (ref).

Cloud metadata (internal/cloudmetadata/)

  • Cloud-agnostic Provider interface with AWS and Azure implementations
  • Wraps re-exported metadataproviders from otel-contrib (no custom IMDS code)
  • Singleton with auto-detection (AWS ? Azure), used by config-downloader and config-translator
  • Replaces ec2util.GetEC2UtilSingleton() in all callers

Cloud provider enum (internal/cloudprovider/)

  • Shared CloudProvider type (Unknown, AWS, Azure)

Re-export wrappers (otel-contrib)

  • resourcedetectionprocessor/metadataproviders/{ec2,azure}/ exposes internal packages for external use

resourcedetection processor translator

  • Added for non-EC2 pipelines (Azure, GCP)
  • Reads append_dimensions and enables only requested attributes
  • Supports OTel-style placeholders (${host.id}, ${azure.vm.size})

ec2tagger changes

  • Removed all volume/disk tagging code (moved to disktagger)
  • Supports OTel placeholders (${host.id}) alongside legacy (${aws:InstanceId})

disktagger processor (plugins/processors/disktagger/)

  • Extracted from ec2tagger as standalone processor
  • AWS: reuses existing volume package (DescribeVolumes + NVMe serial)
  • Azure: IMDS storageProfile (OS + data disks) + /dev/disk/azure/ symlinks with sysfs SCSI fallback
  • Prefix matching for partition names (e.g. sda1 matches sda)

Pipeline translator

  • EC2 mode ? ec2tagger (unchanged)
  • Non-EC2 ? resourcedetectionprocessor
  • disktagger added when disk VolumeId/DiskId dimension is configured

Placeholder resolution

  • Runtime placeholders (${aws:VolumeId}, ${disk.id}) stripped during translation
  • cloudmetadata.GetProvider() replaces ec2util for metadata

Config examples

EC2 (backward compatible):

{"InstanceId": "${aws:InstanceId}", "VolumeId": "${aws:VolumeId}"}

EC2 (OTel style):

{"InstanceId": "${host.id}", "VolumeId": "${aws:VolumeId}"}

Azure:

{"InstanceId": "${host.id}", "Region": "${cloud.region}", "VolumeId": "${disk.id}"}

Known gap

Attribute renaming for non-EC2 clouds: resourcedetectionprocessor adds OTel semantic convention attributes (
e.g. host.id, cloud.region, azure.vm.size) to the Resource. These are not automatically renamed to CloudWatch
dimension names (e.g. InstanceId, Region, VMSize). The resource_to_telemetry_conversion setting copies them to metrics, but with OTel names for CW exporters.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Tested on Ubuntu hosts in AWS and Azure by emitting CPU/MEM/DISK metrics

  • AWS
Screenshot 2026-02-18 at 4 24 58 PM
  • Azure
Screenshot 2026-02-18 at 4 25 05 PM

Requirements

Before commiting your code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

Integration Tests

To run integration tests against this PR, add the ready for testing label.

@movence movence requested a review from a team as a code owner February 18, 2026 21:28
@movence movence force-pushed the hsookim/multi-cloud/metadata branch from 32e08fe to 55df7e2 Compare February 20, 2026 15:47
@movence movence added the ready for testing Indicates this PR is ready for integration tests to run label Feb 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for testing Indicates this PR is ready for integration tests to run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant