Expose the labels package to use with custom stats handler

[eshitachandwani]

Use case(s) - what problem will this feature solve?

Expose GetLabels() function to be used with custom stats handler.

Proposed Solution

Move the labels package from internal to experimental but this is a cross language change and will need to be checked with other language implementations.

Alternatives Considered

Additional Context

[eshitachandwani]

Could you please describe the issue and exact use case for this so that we could discuss possible workarounds and also check with other language implementations before making the change. @seth-epps @davinci26

cc : @dfawley @easwars

[seth-epps]

Our internal grpc-go framework doesn't use open telemetry instrumentation directly and we heavily rely on custom stats.Handler implementations to record metrics, traces, and logs; important for our o11y stack is capturing is the grpc.lb.backend_service rpc label. The motivation to export the Get/Set labels is to effectively initialize those context labels and retrieve them for our internal metrics.

From what I can tell the only way these are added to the context is if the labels are already set/initialized

grpc-go/internal/xds/balancer/clusterimpl/picker.go

Line 100 in 2d2829f

return nil

which a cursory search shows can only be done if the clientMetricsHandler is registered as a stats.Handler using the otel dial instrumentation dial topions. See

grpc-go/stats/opentelemetry/opentelemetry.go

Line 130 in 2d2829f

metricsOpts = append(metricsOpts, grpc.WithChainUnaryInterceptor(metricsHandler.unaryInterceptor), grpc.WithChainStreamInterceptor(metricsHandler.streamInterceptor), grpc.WithStatsHandler(metricsHandler))

grpc-go/stats/opentelemetry/client_metrics.go

Lines 170 to 181 in 2d2829f

	var labels *istats.Labels
	if labels = istats.GetLabels(ctx); labels == nil {
	labels = &istats.Labels{
	// The defaults for all the per call labels from a plugin that
	// executes on the callpath that this OpenTelemetry component
	// currently supports.
	TelemetryLabels: map[string]string{
	"grpc.lb.locality": "",
	"grpc.lb.backend_service": "",
	},
	}
	ctx = istats.SetLabels(ctx, labels)

grpc-go/stats/opentelemetry/client_metrics.go

Line 171 in 2d2829f

if labels = istats.GetLabels(ctx); labels == nil {

Looking at the java implementations the optional labels are private on the otel class and all the tracers seem to be package private so exposing the labels directly here would move us away form that. It also seems like it's very much isolated to the open telemetry packages. Poking around the core implementation it does seem somewhat more relaxed in letting the stats implementation handle the labels, though I will admit my ability to dig through s is restricted by my lack of experience with the packages so I'll defer to others to confirm what may or may not be publicly accessible.

I wonder if there's a minimal refactor that could achieve something that delegates the label tracking to a component rather than setting it directly on the context 🤔

[davinci26]

+1 on @seth-epps (also if it is wasnt obvious context, me and Seth are working on the same team at Reddit)

[mbissa]

Can you specify where exactly do you need the backend service label?

[seth-epps]

We collect metrics using a custom stats.Handler so a stripped down example might be something like this

func initTelemetryLabels(ctx context.Context) context.Context {
	var labels *estats.Labels
	if labels = estats.GetLabels(ctx); labels == nil {
		labels = &estats.Labels{
			TelemetryLabels: map[string]string{},
		}
		ctx = estats.SetLabels(ctx, labels)
	}
	return ctx
}

func (st *Handler) TagRPC(ctx context.Context, stat *stats.RPCTagInfo) context.Context {
	// initialize the stats labels if not already set by another handler
	ctx = initTelemetryLabels(ctx)
	
	// do more stuff to tag tpc
	return ctx // or context.WithValue(...)
}

func (st *Handler) HandleRPC(ctx context.Context, stat stats.RPCStats) {
	lbService := "unknown"
	labels := estats.GetLabels(ctx)
	if labels != nil {
		if service, ok := labels.TelemetryLabels["grpc.lb.backend_service"]; ok {
			lbService = service
		}
	}
	switch s := stat.(type) {
	case *stats.Begin:
		// these are placeholder metrics
		clientActiveRequests.With(
			prometheus.Labels{
				"lb_service": lbService,
			}
		).Inc()
	case *stats.End:
		clientActiveRequests.With(
			prometheus.Labels{
				"lb_service": lbService,
			}
		).Dec()

		clientTotalRequests.With(
			prometheus.Labels{
				"lb_service": lbService,
			}
		).Inc()
	}
}

Which effectively gives us a client request distribution surfaced in our metrics since we don't necessarily have the otel instrumentation needed to get the labels there. This is particularly important for us because the service-to-service communication platform relies on xds where a given listener can have multiple routes and multiple clusters.

[mbissa]

Scheduled for end of Q1 2026.

[seth-epps]

If there's any additional context around the support this proposal I'd be happy to contribute and drive it forward. It's a bit unclear right now so I'm trying to plan accordingly

[davinci26]

Hey folks,

I wonder if there has been any update because we find needing this feature. Also I want echo again Seth's point that we are happy to contribute/help with this in any way we can. So if you want/can move some of the conversation to the public (or even via a VC/meeting) we would be more than happy to help out based on your guidance!

cc @mbissa @dfawley @easwars

[mbissa]

@seth-epps @davinci26 - We had a discussion within the grpc-go team.

We cannot expose the ability to Set or modify the internal Labels data structure directly, as allowing external modification of the context state poses safety and compatibility risks.

However, we are supportive of adding a function (in the experimental/stats package) that fetches these values as strings.

Note: Since you aren't using the OpenTelemetry handler (which is what normally initializes these labels at client_metrics.go L165), the internal picker logic would likely need to be updated to automatically initialize the labels container if it is missing.

I am marking this as 'help wanted' to implement this retrieval function and the necessary internal adjustment in the picker. Please let me know if you are interested in working on this.