gRPC: expose API to receive full-copy labels

[giulianopz]

Is your proposal related to a problem?

ATM, the gRPC client still uses a deprecated module (i.e. gogo/protobuf) to register a custom type that allows to zero-copy labels. This makes the Query and QueryRange API completely unsafe for consuming labels along with samples.

Describe the solution you'd like

Completely remove the ZLabel trick or expose a new safe API along with the existing one.

Describe alternatives you've considered

Even forcing a new allocation by cloning label strings with strings.Clone I was not able to safely consume these labels, since I sporadically observe failures (say 1 out of 10k runs) where characters in labels are mangled later on during the result processing. This must be because somehow the GC still holds a reference to the shared buffer used by the gRPC decoder. I'm pretty sure there are no dangling references in my code, but I was not able to determine where the error might be in the Thanos gRPC client.

Additional context

The limits of socalled 'zlabels' are well-documented in the code:

• thanos/pkg/store/labelpb/label.go

  Lines 116 to 121 in 7488f83

  	// ZLabel is a Label (also easily transformable to Prometheus labels.Labels) that can be unmarshalled from protobuf
  	// reusing the same memory address for string bytes.
  	// NOTE: While unmarshalling it uses exactly same bytes that were allocated for protobuf. This mean that *whole* protobuf
  	// bytes will be not GC-ed as long as ZLabels are referenced somewhere. Use it carefully, only for short living
  	// protobuf message processing.
  	type ZLabel Label

• thanos/pkg/store/labelpb/label.go

  Lines 41 to 43 in 7488f83

  	// ZLabelsFromPromLabels converts Prometheus labels to slice of labelpb.ZLabel in type unsafe manner.
  	// It reuses the same memory. Caller should abort using passed labels.Labels.
  	func ZLabelsFromPromLabels(lset labels.Labels) []ZLabel {

• thanos/docs/internals/buffers-guide.md

  Line 15 in c50f95a

  We still use gogoproto so in the protobuf decoder we specify a custom type for labels - ZLabels. This is a "hack" that uses unsafe underneath. With the `slicelabels` tag, it is possible to create labels.Labels objects (required by the PromQL layer) and reuse references to strings allocated in the protobuf layer. The protobuf message's bytes buffer is never recycled and it lives as far as possible until it is collected by the GC. Chunks and all other objects are still copied.

[aviralgarg05]

I would like to solve this issue

[GiedriusS]

I have trying to solve this problem over the years in various ways and I came to the conclusion that we should directly construct labels.Labels in the protobuf unmarshal function. I started this repo https://github.com/thanos-io/protoc-gen-go-thanos but no code so far. https://giedrius.blog/2022/02/04/things-learned-from-trying-to-migrate-to-protobuf-v2-api-from-gogoprotobuf-so-far/. Have you seen the previous attempts?

Another thing that we should explore is implementing cap 'n proto interfaces for our others APIs. See #7071 discussion.

If you have time and energy, we could collaborate to solve this once and for all.