Export detailed metrics via OpenTelemetry

[curiousleo]

Is your feature request related to a problem?

grpc-java exports detailed metrics via OpenCensus (code in /census). There's a guide for how to access those metrics here: https://opencensus.io/guides/grpc/java/.

OpenTelemetry is the official successor to OpenCensus. But grpc-java does not expose metrics in an OpenTelemetry-compatible way.

Describe the solution you'd like

It would be nice if the detailed metrics currently exposed via OpenCensus were exposed in an OpenTelemetry-compatible way.

Describe alternatives you've considered

Setting up an OpenCensus-based metrics pipeline just for grpc-java -- certainly possible, but runs counter to the idea of unifying metrics and tracing under the OpenTelemetry umbrella.

Additional context

Given that OpenTelemetry is the official successor to OpenCensus, I tried to figure out if OpenTelemetry is OpenCensus-compatible, but couldn't find relevant documentation. It is possible that this is the case and that OpenTelemetry support is a no-op as a result.

[sanjaypujare]

@ericgribkoff may be this is for you?

[ericgribkoff]

OpenTelemetry is the official successor to OpenCensus, but it has not yet hit it's v1.0 release for Java: they are currently at v0.8. When OpenTelemetry was originally announced (e.g., here) the intent was for OpenTelemetry to be backwards-compatible with OpenCensus, meaning that gRPC's existing OC support would continue to work:

When each language reaches parity, the corresponding OpenTracing and OpenCensus project will be sunset. This means that the old projects will be frozen, but the new project will continue to support existing instrumentation for two years, via a backwards compatibility bridge.

I don't see a mention of this on the OpenTelemetry docs anymore, so it's not clear that they still plan the backwards-compatibility bridge. It does not exist as of their v0.8 release.

You currently can use OpenTelemetry with gRPC Java for tracing, either as shown in their "advanced example" (https://github.com/open-telemetry/opentelemetry-java/tree/master/examples/grpc) or automagically using their auto-instrumentation package (https://github.com/open-telemetry/opentelemetry-java-instrumentation, this may be limited to clients-only if I recall correctly). I have recently used their tracing with gRPC Java and it seems to work well (just for a demonstration; prod use would need to check with the OpenTelemetry team about their recommendation regarding using a pre-v1.0 release).

My overall understanding of OpenTelemetry is that they are still working on part of the specification for metrics (specifically, views), so the metrics implementations may be somewhat behind their tracing support, depending on the language. It looks like OpenTelemetry's Java implementation actually has some support for views already, but this would be ahead of the spec and so I haven't looked too deeply into it yet, as it would seem to be subject to change.

We do want to support OpenTelemetry metrics and tracing with gRPC Java in the future. It still remains to be seen if this will take the form of a package under the io.grpc namespace, like we did with OpenCensus, or will be in another repository similar as with their existing tracing interceptor in the auto-instrumentation OT package. I should note that there's no need for tracing or metrics to be part of the gRPC library itself, as the mechanism the gRPC OpenCensus instrumentation uses to instrument calls are themselves exposed API (with the caveat that these are low-level and intended explicitly for other libraries, not end-users, to consume). Part of the mistake that was made with OpenCensus, and part of the motivation for having it be part of the gRPC library itself originally, was that we wanted to automatically enable OpenCensus for gRPC whenever it is present on the classpath. We will want to avoid that for OpenTelemetry (see #5624 for some of the difficulties this has led to with OpenCensus)

Sorry for not having a clear and definite answer as to when OpenTelemetry support will be available: we've been mostly in a "wait-and-see" pattern as OT moves towards its v1.0 release. The current recommendation would be to stick with OpenCensus for metrics (and, therefore, probably tracing as well - OC can be configured to use a traceformat supported by OpenTelemetry, but that configuration knobs does not exist for gRPC+OC). We should have a better story for all this once OT hits v1.0 (assuming v1.0 includes a complete metrics implementation, which I assume it will but have not checked their roadmap recently).

[ericgribkoff]

I'm going to mark this closed since there are no pending work items for us to address here. Once OT has a v1.0 release we can revisit this. (@curiousleo if there's anything I've missed let me know and I can reopen).

[curiousleo]

Hey @ericgribkoff, thank you very much for the thoughtful response!

The main point, as I understand it, is that if the OpenTelemetry 1.0 spec and Java implementation cover metrics, once it comes out, the gRPC team will seriously consider making gRPC metrics available via OpenTelemetry in some way. That's really good to hear.

Just a note on the auto-instrumentation in OT as of now:

automagically using their auto-instrumentation package, this may be limited to clients-only if I recall correctly

For tracing, the server side is covered too by the Java+gRPC+OpenTelemetry auto-instrumentation.