[libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor by Kavindu-Dodan · Pull Request #41636 · elastic/beats

Kavindu-Dodan · 2024-11-13T19:52:35Z

Proposed commit message

This PR fixes incorrect recognition of EC2/AWS cloud provider as Openstack. The root cause was the common metadata endpoints used by both AWS SDK & Openstack logic. And this happened when IMDSv2 is disabled in AWS.

I attempted to migrate Openstack logic to another metadata implementation. However, I did not manage to create a fully functioning setup to validate the implementation. Hence, this PR focuses on a priority-based solution where priority is given for SDK-backed metadata fetching over HTTP endpoints.

Current priory providers are - aws/ec2 & azure

Note - I have done a minor refactoring to rename Local struct property to DefaultEnabled to make intention clearer

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

You need a local copy and an EC2 instance to validate the fix.

Enable metadata service in EC2 instance and make IMDSv2 optional
Build a beats (ex:- metricbeat) module based on this libbeat change
Copy the beats module to EC2 instance and start the module with add_cloud_metadata processor enabled & logs set to debug for more in-depth logs & no provider enforced
Observe logs and see data (ex:- system monitoring) through Kibana to validate cloud provider detection

Related issues

Closes add_cloud_metadata detecting wrong cloud provider (aws as openstack) #13816

Screenshots

-IMDSv2 disabled

Processor enabled but no provider enforced

Debug logs on multi-result and priority based selection

Cloud provider detected correctly,

elasticmachine · 2024-11-13T19:53:00Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

kaiyan-sheng

Thanks for working on this issue! Can we document this new logic around priorityProviders in the doc please?

Kavindu-Dodan · 2024-11-14T17:52:54Z

Thanks for working on this issue! Can we document this new logic around priorityProviders in the doc please?

Thank you, I added documentation :)

axw · 2024-11-18T01:05:16Z

@Kavindu-Dodan with this change could we end up having the opposite problem, where OpenStack is misidentified as AWS?

Kavindu-Dodan · 2024-11-18T17:24:57Z

@Kavindu-Dodan with this change could we end up having the opposite problem, where OpenStack is misidentified as AWS?

This shouldn't happen as priority is given for cloud provider SDK based metadata fetchers (AWS & Azure ¹), where they should fail if configurations are not set (ex:- AWS credential profile/env vars).

https://github.com/elastic/beats/pull/41636/files#diff-54e31882d706ede511023836a842dd7c1e570ce1eab2c7208601afacee49a995R76-R79 ↩

CHANGELOG.next.asciidoc

libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.asciidoc

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

…a processor (#41636) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641) # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go

…a processor (#41636) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641)

…a processor (#41636) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641) # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go

…a processor (#41636) (#41815) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641) Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…a processor (#41636) (#41814) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641) # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…a processor (#41636) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641)

…a processor (#41636) (#42606) * rename misleading variable Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * introduce provider priority Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go * isolate priority logic and add testing Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * documentation Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * review changes Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> (cherry picked from commit 6d4e641) Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

Kavindu-Dodan added bug backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify labels Nov 13, 2024

Kavindu-Dodan requested review from a team, kaiyan-sheng and zmoog November 13, 2024 19:52

Kavindu-Dodan requested a review from a team as a code owner November 13, 2024 19:52

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 13, 2024

Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Nov 13, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 13, 2024

mergify bot assigned Kavindu-Dodan Nov 13, 2024

kaiyan-sheng reviewed Nov 14, 2024

View reviewed changes

Kavindu-Dodan requested a review from a team as a code owner November 14, 2024 17:51

Kavindu-Dodan requested review from belimawr and faec November 14, 2024 17:51

Kavindu-Dodan force-pushed the fix/fix-aws-openstack-metadata-conflict branch from adcdc7d to cbb6a96 Compare November 14, 2024 17:52

Kavindu-Dodan requested a review from kaiyan-sheng November 14, 2024 22:55

Kavindu-Dodan mentioned this pull request Nov 14, 2024

add_cloud_metadata detecting wrong cloud provider (aws as openstack) #13816

Closed

belimawr reviewed Nov 18, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

CHANGELOG.next.asciidoc Show resolved Hide resolved

libbeat/processors/add_cloud_metadata/docs/add_cloud_metadata.asciidoc Outdated Show resolved Hide resolved

Kavindu-Dodan force-pushed the fix/fix-aws-openstack-metadata-conflict branch from 5b6ab95 to f416516 Compare November 19, 2024 15:22

elastic deleted a comment from mergify bot Nov 19, 2024

Kavindu-Dodan requested a review from belimawr November 19, 2024 15:24

belimawr approved these changes Nov 19, 2024

View reviewed changes

Kavindu-Dodan added 3 commits November 22, 2024 08:16

rename misleading variable

9b2e185

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

introduce provider priority

c6d64c4

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> # Conflicts: # libbeat/processors/add_cloud_metadata/providers.go

isolate priority logic and add testing

cf002ea

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan added 2 commits November 22, 2024 08:16

documentation

9ba0982

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

review changes

69c70b2

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the fix/fix-aws-openstack-metadata-conflict branch from f416516 to 69c70b2 Compare November 22, 2024 16:16

kaiyan-sheng approved these changes Nov 26, 2024

View reviewed changes

Kavindu-Dodan merged commit 6d4e641 into elastic:main Nov 27, 2024

This was referenced Nov 27, 2024

[8.16](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor #41814

Merged

[8.x](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor #41815

Merged

Kavindu-Dodan added the backport-8.17 Automated backport with mergify label Feb 5, 2025

mergify bot mentioned this pull request Feb 5, 2025

[8.17](backport #41636) [libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor #42606

Merged

6 tasks

MichaelKatsoulis mentioned this pull request Feb 18, 2025

[add_cloud_metadata] Superfluous error logging #38679

Closed

khushijain21 mentioned this pull request Jun 23, 2025

test4 #44958

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor#41636

[libbeat] fix: aws & openstack metadata conflict in add_cloud_metadata processor#41636
Kavindu-Dodan merged 5 commits intoelastic:mainfrom
Kavindu-Dodan:fix/fix-aws-openstack-metadata-conflict

Kavindu-Dodan commented Nov 13, 2024 •

edited

Loading

Uh oh!

elasticmachine commented Nov 13, 2024

Uh oh!

kaiyan-sheng left a comment

Uh oh!

Kavindu-Dodan commented Nov 14, 2024

Uh oh!

axw commented Nov 18, 2024

Uh oh!

Kavindu-Dodan commented Nov 18, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

Kavindu-Dodan commented Nov 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Nov 13, 2024

Uh oh!

kaiyan-sheng left a comment

Choose a reason for hiding this comment

Uh oh!

Kavindu-Dodan commented Nov 14, 2024

Uh oh!

axw commented Nov 18, 2024

Uh oh!

Kavindu-Dodan commented Nov 18, 2024

Footnotes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Kavindu-Dodan commented Nov 13, 2024 •

edited

Loading