Allow user to disable certain cloud metadata providers

[jordansissel]

Describe the enhancement:

There are two (maybe more?) add_cloud_metadata providers which will talk to remote addresses: tencent and alibaba. Everything else appears to talk to an internal/private address (169.254.169.254)

In order to minimize unexpected network activity, an operator should be able to disable cloud metadata providers which talk to external services. For example, an operator who knows they are running on Azure would know that querying Tencent's metadata doesn't make sense, and should be able to disable it, especially as it is an external query.

Describe a specific use case for the enhancement or feature:

There are security concerns with respect to nodes announcing themselves to external entities. We have at least one user who reported suspicious network activity (unexpected external http calls) that was tracked back to beats add_cloud_metadata.

[andrewkroh]

+1

IMO we should disable by default the providers that are not using hard-coded link local IP addresses. If you are running on those platforms you will need to enable them in your config.

[jordansissel]

+1 disabling by default. I haven't checked, but if no cloud_metadata providers are found, does beats log a warning? Such a message could help users on Tencent and Alibaba to remind them that these providers need to be enabled explicitly.

[weltenwort]

Just as a note: AFAIK the Infrastructure UI relies on the cloud_metadata processor being enabled for some of its host grouping options. Changes to the default availability of these metadata should probably be reflected in the infrastructure monitoring installation documentation.

[tsullivan]

Hi, I was pointed to this issue when I was warned by Little Snitch that metricbeat was sending requests to 100.100.100.200

I suppose it's just being labelled as a "suspicious process" because Little Snitch doesn't know what metricbeat is.

Use case:

• Running on MacOS
• Built metricbeat 8.0.0 from source
• Run metricbeat directly on the command line

I thought I would add this detail here, but I'm not up to speed on the issue to know whether the request made is intentional or not. I'm not running on a cloud provider, so I suspect it is not.

[jordansissel]

Changes to the default availability of these metadata

I believe the current proposal is to by-default disable cloud providers which are require external calls (alibaba + tencent) and that other providers (aws, gcp, etc) will remane enabled by default with add_cloud_metadata.

@weltenwort does this help resolve your concerns? Some documentation changes will be necessary to mention this special casing, but I feel that's implied as part of our general change processes.

[weltenwort]

yes, all good 👍 I just wanted to establish the mental cross-reference for anyone working on this.

[ruflin]

+1 on this approach.

Today we don't have any config options to select which providers should be run. To make this a non breaking change for tencent / alibaba users in 7.0 we could introduce such a config and set aws, gcp as defaults in our config. If the providers config is missing, it would still try all.

Our default config would change to (or similar):

processors:
- add_cloud_metadata:
    providers: ["aws", "gcp"]

@andrewkroh Remove the :infrastructure label as I think this falls under core

[alastairs]

So we're four minor releases into the 7.x lifecycle and this issue is still open. This raised concerns for us too (see my forum post, from where I was redirected to this issue). As far as we can tell, it's only metricbeat making these calls, not filebeat or packetbeat (which we are also running), which seems curious. Please could someone confirm whether there is a timeline decided for fixing this issue?

[urso]

This issue is currently worked on in: #13812

This introduces a providers setting, as proposed by @ruflin. But after debating the 'default' with myself a little I opted to introduce a breaking change and disable Alibaba/Tencent by default thinking: Calling a potential remote should be an opt-in, not an opt-out.