Making tracing SDK metrics aware

[sfriberg]

Making Tracing API metrics aware

One interesting aspect of Open Telemetry is the goal to provide tracing, metrics, and later a logging API. Right now these APIs are fairly separate so as an end-user or library owner I would need to write code to add a span and some more code to add a metric. This is not different that I would need to do today with OpenCensus or with OpenTracing and a metrics library (Prometheus, DropWizard, etc.), but I think OT would allow this to be simplified due to it's all in one nature.

With the metrics and API that takes a Measurement it feels like the integration with Spans would come fairly natural.

Why

Single code point extension, get metrics for "free".

As a end-user I want to write a little code as possible. Instead of couple of lines for metrics, a few more for tracing and one for logging, if I can get it all done in a single place that would be helpful.

Increased uptake of metrics library

Easily getting metrics from spans would be one further path that we could increase the uptake of the metrics library. Switching from OpenCensus and OpenTelemetry is pretty much required for tracing, but metrics doesn't have the same forced migration and will face a much more mature and larger ecosystem.

Span exploration

Currently the naming of Spans is rather close to that of using labels in metrics names, it ends up being an explosion in names that all have a different path, instead of having a HTTP request span that you can easily filter for properties you are interested in.

Filtering and sampling of Traces and Spans using metric data

One of the longer term interesting aspects of this closer relationship will be that metrics could be used by spans/traces do decide if a span should be after the trace is complete/tail-based sampling. Since the metrics name is know (or the same as the span name) as well as labels used for the metric you would be able to easily query a metrics db to get for example the 99%tile for the last 24 hour and explicitly store all of those metrics as well as any randomly sampled traces.

How

Others will probably have better suggestion, but an initial idea would be to have the constructor/builder pattern simply allow, generateMetric(boolean) as part of building the Span.

As for labels I think it would be interesting to discuss if labels should be part of the span. One could think of it as being different layers of contextual data for a span. Top layer would be Resources (process level labels), Labels (general request labels), and Attributed (request unique (or high cardinality) labels). Metrics and spans would then be exported with these as appropriate.

Naming of spans could be the same as the metric for easy association and lookup, but for backwards compatibility we could allow a Operation Name the constructed through a templated string that could read a label, such as {{label.method}}:{{label:parameterized_url}} for a HTTP request span.

[jmacd]

The tracing API should not be made aware of metrics. A tracing SDK should be made configurable to support the kinds of extension you are interested in. The API already supports what is needed in this regard -- every metrics API has a context argument, and an integrated SDK could be configured to aggregate metrics on a per-span level. Issue #259 is a placeholder for this sort of discussion while #347 is still in-progress.

This topic was discussed in today's Spec SIG meeting as well. I am looking for a recording.

[sfriberg]

@jmacd Why do you see this more as a SDK integration than part of the API?

Similar to Resources being shared between the two, why couldn't labels be a shared API construct in the same fashion?

My understanding of the API vs SDK separation isn't the best either, any pointers to documentation that defines them?

[jmacd]

We have a Library Guidelines document that explains the API/SDK separation. The way I think about this is that the API should define semantics of the operations and leave implementation to the choice of SDK. The SDK should be able to implement metrics-aware-tracing and tracing-aware-metrics if it pleases....

So long as we set up the APIs with the necessary information crossing the API/SDK boundary and give useful semantics to the API operations.

For tracing-aware-metrics, the task would be to develop a SDK that implements both tracing and metrics with the awareness that you want.

There are several natural metrics implied by every span that could be automatically instrumented. There's a counter instrument for the number of span started, a measure instrument for the latency of each span, and in some cases it's useful to think of a distinct counter of the number of spans finished.

Basically, I think we can accomplish the sorts of things you imagine without any new API surface. You had written about a generateMetric(boolean) method on the Span API. I'm proposing that this can be accomplished by a metrics-or-tracing configuration in the SDK. If there is enough interest, we should be able to get features of this nature into the default SDK.

[kbrockhoff]

I agree this should be handled by the SDK. I would really like to see this in the default SDK.

[sfriberg]

I think to make this feasible and "easily" implementable in the SDK I think updating the API is required to a certain extent.

With the rather specialized Span name that is currently being used in Distributed Tracing, such as "GET:/resource/{id}", using a more generic name similar to a metric name would be useful. A HTTP request could have the generic Span name "http_request", from this we could automatically generate metrics as required, "http_request_duration (histogram)", "http_request_status (counter)". We could generate these through SDK configuration and specify which data to extract from a Span to include in the metric, and which labels.

For backwards compatibility we could allow a Span to have a "OperationName" format string that would be a template string, "{{label.method}}:{{label.path}}", that picks up different labels from the Span to build the name. This could potentially also just be a SDK configuration or exporter configuration so that it is only actually generated when required. This would also help solve one of the issues we ran into with OpenTracing where operation name is set when the span is created so to change it contribution libraries added special hooks to allow users to change operation name and other labels as part of the span creation[1][2].

I think the other interesting API change would be to try and unify the resource/label-set/attributes that are current being used for process/metrics/spans, and instead use a single type that can be assigned to every level. If we have a LabelSet that can handle merges, we can use it to be able to add labels to a Resource, TraceFactory, NamedTracer, NamedMetricsProvider (not sure what the real name is), Metric and Span. The would be merged from highest to lowest level with a a defined precedence of what labels are being kept/overwritten. The SDK can then be used to define what labels to drop, keep etc from Spans and Metrics, and a user of the API and/or the SDK will have a single construct to learn and understand instead of 4 different varieties of sets.

1 - https://github.com/opentracing-contrib/java-web-servlet-filter/blob/master/opentracing-web-servlet-filter/src/main/java/io/opentracing/contrib/web/servlet/filter/ServletFilterSpanDecorator.java
2 - https://github.com/opentracing-contrib/java-jaxrs/blob/master/opentracing-jaxrs2/src/main/java/io/opentracing/contrib/jaxrs2/client/ClientSpanDecorator.java

[iNikem]

@sfriberg Do you I understand your proposal (or at least part of it) correctly:

• Span name should be low cardinality, e.g. http_request, and suitable for metric name
• Span should have a separate attribute, called OperationName, to be dynamically composable from other attributes

If yes, then below are my objections :)
First, Otel Collector already allows for span name composition based on arbitrary labels.
Second, if current span names are too high cardinality for your metrics, why cannot you use any other attributes/labels for metric name generation? Both of the proposals above seems to me like pushing to already existing functionality lower in the stack, to the actual data generation. I would argue that collector and/or backend are better places for them. As it is currently :)

[sfriberg]

Thanks for the feedback @iNikem. Yes, that resembles what I'm looking at

First, Otel Collector already allows for span name composition based on arbitrary labels.
Didn't know this, do you have a pointer?

The main reason is to make it easier for developers and users of the metric to know what to look for and be able have some control of what they want to be collected. A span might be sampled so not making it to the backend and thus getting a completely accurate metric won't be possible unless you have a metric as well. So in part yes it is pushing functionality further down in the stack, but mainly to allow easier implementation for a developer that instead of adding a span + two counters (total requests, total duration) that would be created automatically with a easily found name.

[iNikem]

@sfriberg https://github.com/open-telemetry/opentelemetry-collector/blob/master/processor/README.md#name-a-span

[sfriberg]

@iNikem Thanks, had missed that. Is this something we purely aim to do in the collector or is the intent to support it at all levels in SpanProcessors?

Having this capability feels like it would further warrant a basic naming scheme on the generation side so that you could easily write a rule for the specific span you are interested in using the name. If the span names comes with complex naming doing the filtering and renaming gets harder.