`github.com/golang/protobuf/proto` is deprecated

[ctlong]

This project should switch to using google.golang.org/protobuf/proto instead.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[ctlong]

In order to use google.golang.org/protobuf/proto marshaling, we need protobuf files that include reflection. I thought that regenerating the .proto files located in https://github.com/cloudfoundry/loggregator-api would be enough to facilitate the switch to using google.golang.org/protobuf/proto, but it turns out that those are just the loggregator_v2 photobuf files. The loggregator_v1 protobuf files live in https://github.com/cloudfoundry/sonde-go, and will also need to be updated.

[ctlong]

Currently the steps to resolving this issue appear to be:

• Update loggregator_v2 protobuf files
  1. Update loggregator-api where appropriate
  2. Regenerate loggregator_v2 protobuf files from loggregator-api and merge them into go-loggregator
• Update loggregator_v1 protobuf files
  1. Update dropsonde-protocol where appropriate
  2. Regenerate dropsonde-protocol protobuf files and merge them into sonde-go
  3. Pull the new sonde-go protobuf files into go-loggregator
• Remove all usage of github.com/golang/protobuf/proto from go-loggregator

Open question:

• Why do we use gogo/protobuf to generate many of our protobuf files? Can we remove our dependency on that too?

[mkocher]

Note for future self and @ctlong:

the modern magic incantation for generating these is something like this:

OUT_PATH="Menvelope.proto=rpc/loggregator_v2,Mingress.proto=rpc/loggregator_v2,Megress.proto=rpc/loggregator_v2:."
protoc egress.proto ingress.proto envelope.proto   -I /tmp/foo3/loggregator-api/v2 --go_out=$OUT_PATH --go-grpc_out=require_unimplemented_servers=false,$OUT_PATH

the require_unimplemented_servers part is necessary to make it work though not advised these days, I didn't read enough to understand why yet.

[ctlong]

I made a repo to help compare the old generated code for loggregator v1 against the new generated code for loggregator v1: https://github.com/ctlong/loggregator-v1-protobuf-example. And another for v2: https://github.com/ctlong/loggregator-v2-protobuf-example

This is to test backward compatibility as we move forward with these changes. We definitely don't want to break old clients and old servers where possible.

This is also to benchmark our marshaling and unmarshaling to get an idea an idea of how our code's performance may improve.

[ctlong]

I think that we should move away from using the github.com/gogo/protobuf/proto module.

The benchmarks tests for our v1 and v2 rpc code seem to show that our fastest option for go marshaling/unmarshaling in v1 is to use github.com/gogo/protobuf/proto to generate our go code and then use that library for all marshaling/unmarshaling. google.golang.org/protobuf gets close though. In v2 google.golang.org/protobuf is fastest.

$ cd loggregator-v1-protobuf-example && go test -bench=. ./...
goos: darwin
goarch: amd64
pkg: test
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkNewProtoMarshal-16          	 8778264	       129.9 ns/op
BenchmarkNewProtoEasyMarshal-16      	 7080901	       164.7 ns/op
BenchmarkNewProtoGogoMarshal-16      	 6467515	       177.1 ns/op
BenchmarkNewProtoUnmarshal-16        	 7396680	       161.1 ns/op
BenchmarkOldProtoMarshal-16          	  458138	      2520 ns/op
BenchmarkOldProtoEasyMarshal-16      	 7009828	       166.4 ns/op
BenchmarkOldProtoGogoMarshal-16      	20870960	        54.19 ns/op
BenchmarkOldProtoUnmarshal-16        	  195733	      5692 ns/op
BenchmarkOldProtoEasyUnmarshal-16    	 2607628	       463.9 ns/op
BenchmarkOldProtoGogoUnmarshal-16    	13195027	        84.40 ns/op

$ cd loggregator-v2-protobuf-example && go test -bench=. ./...
goos: darwin
goarch: amd64
pkg: test/cmd/new
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkNewProtoMarshal-16      	 4742690	       246.9 ns/op
BenchmarkNewProtoUnmarshal-16    	 2956844	       397.0 ns/op
goos: darwin
goarch: amd64
pkg: test/cmd/old
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
BenchmarkOldProtoMarshal-16          	  465183	      2211 ns/op
BenchmarkOldProtoGogoMarshal-16      	 1000000	      1103 ns/op
BenchmarkOldProtoUnmarshal-16        	  152378	      7218 ns/op
BenchmarkOldProtoGogoUnmarshal-16    	 1654905	       730.5 ns/op

However, github.com/gogo/protobuf/proto is currently unsupported, looking for new ownership, and hasn't had a release for close to a year. The skippy peanut butter CVE that library opened us up to took a while to notice and was a pain to fix. In contrast, the maintainers of google.golang.org/protobuf are quite active, and that project appears to have incorporated the best parts of github.com/gogo/protobuf/proto over the last few years.

It seems like a better idea to standardize around a library that definitely has a future.