OM 2.0: Federation: allow distinguishing classic histograms and NHCB

[krajorama]

Problem Statement

From @MichaHoffmann

Assume two (or more) sources for federation. One sends Prometheus classic histograms and the other sends native histograms with custom bucket (NHCB).

Currently OpenMetrics does not allow to expose NHCBs , thus the federated endpoint will show all as classic histograms. This is a loss of information.

Proposed solution

<--! describe the high-level outline of your proposed solution. If the overall direction is accepted and needs more detail, a complete proposal can be written to https://github.com/prometheus/proposals-->

Alternatives considered

<--! describe the high-level outline of alternative solutions you considered, and why you chose your proposed solution over these alternatives-->

1. Current state, do not support.

[bwplotka]

Currently OpenMetrics does not allow to expose NHCBs , thus the federated endpoint will show all as classic histograms. This is a loss of information.

It's not if you treat NHCB as storage detail, not PromQL change.

Like, I wonder if you want to leak this abstraction or should we rather work on prometheus/prometheus#17147

[dashpole]

From @MichaHoffmann

its a problem because we get something different during federation then what we scrape in the target prometheus by default, which is weird and also dangerous since we get 20x the series from federation we have in the target prom

From @bwplotka

in fact OpenMetrics 2 will ONLY have nhcbs

I would prefer if we thought about it the way @bwplotka suggested. The OM 2.0 spec will have only "composite" histograms, as histograms do not contain any series with "le" labels, or with _bucket, _sum, or _count suffixes.

The more interesting question is how this interacts with the current Prometheus server's configuration options: always_scrape_classic_histograms and convert_classic_histograms_to_nhcb. We have two options:

1. Keep the same ingestion behavior as OM 1.0. This means that by default we still convert OM 2.0's "composite" histograms into classic histograms, and users can use convert_classic_histograms_to_nhcb and always_scrape_classic_histograms the same way they do today.
2. [preferred] Treat OM 2.0 histograms as NHCB by default. convert_classic_histograms_to_nhcb would have no effect, and always_scrape_classic_histograms would allow you to get the classic form back. Maybe we want an option for only scraping classic too? Edit: This is what @bwplotka pointed to in nhcb: Option to unroll nhcbs to classic histograms for RW (common function) prometheus#17147.

[bwplotka]

For clarity:

Michael's case: 

• new histogram added, scraped with NHCB on edge
• global prom automatically catches it, but does not convert to NHCB, because PromQL is different.
• Boom, global prom can't globally convert due to PromQL difference, can't convert by series either.

I think, it deserves bigger design on all cases.

We have two big directions to decide on:

A) Manual flexibility on both ends:

• input (what to convert what not)
• output (all classic or nhcb as nhcb query modeVS all classic or nhcb as classic query mode)

B) Trying to maintain what it should be "queried as" from the very source. Like imagine prom -> collector -> zone prom -> global prom -> thanos   chain or along the lines.  This means either some metadata or NHCBs in exposition formats for federation problem and... OTLP? IMO that one is hard as it will delay all migrations of classic histograms to ~decades and the middleman can't reduce their storage resources by ~30% or more with NHCBs (and make it atomic). OpenTelemetry already cuts this information too (all is packed into Otel explicit histogram).

From @dashpole options: 1 follows B, 2 follows A 🙃