Only collect usage data once a day from kibana monitoring

[chrisronline]

Relates to #32519

This is a straight copy and paste from this comment. This PR implements this idea.

In order for this PR to work, you'll need to update the .monitoring-kibana index template in ES.

PUT _template/.monitoring-kibana

Copy and paste the json from here -> https://gist.github.com/chrisronline/fdf4f549d4e96cdda06e7b20cf88ce2a

Okay I think I have a plan for this and want to run it by everyone. This solution will ensure that kibana monitoring collection will only poll for usage collector data at the same interval as the normal telemetry service.

• Add a flag (isUsageCollector) to each usage collector, as an actual property similar to ignoreForInternalUploader so monitoring can know it's a usage collector (see why we have to do it this way)
• Modify the kibana monitoring collection code to ONLY collect from usage collectors at the same rate as default telemetry (This will be kept in sync by exposing the interval through the xpack_main plugin). However, the monitoring code will pull from stats collectors at the current interval (default to 10s) to ensure the monitoring service and UIs continue to function properly. This will mean that some .monitoring-kibana-* documents will contain a usage field, and some will not.
• Update the .monitoring-kibana index template to include mappings for the usage field (it currently does not) - Doing this will require a version dance as we want to ensure that folks don't run into a situation where Kibana is assuming this mapping exists but it doesn't yet in ES. This shouldn't happen as we explicitly recommend the monitoring cluster to be at least the same version as all monitored clusters
• Update the query for collecting usage data from monitoring indices for telemetry to filter out documents that do not contain the usage field to ensure it only fetches documents that actually contain usage data (which will now occur at the same interval as default telemetry)

[elasticmachine]

💔 Build Failed

• continuous-integration/kibana-ci/pull-request