upgrade to K8s 1.35 (K8s Go client 0.35)

[reggie-k]

Summary

One breaking change I have identified is:

• Kubernetes API Go types were not suitable for use with the google.golang.org/protobuf library, and no longer implement ProtoMessage() by default to avoid accidental incompatible use.

https://github.com/kubernetes/kubernetes/blob/5151f58ef08d9fb26931019d6d844292aeb57637/CHANGELOG/CHANGELOG-1.35.md?plain=1#L259

In my understanding, we are supposed to have a wrapper of K8s APIs (for ListEvents, ListResourceEvents) and return that from Argo CD API cause we can no longer return K8s API types directly (they are not protobuf message type compatible anymore).

For 1.35, this is a deprecation notice, the support will be removed in 1.36.

Looks like we can use go build -tags kubernetes_protomessage_one_more_release and then delay the wrapper creation till the upgrade to K8s 1.36, but not sure what upgrade path is better.

[brantgurga]

The .status.terminatingReplicas field not being declared in schema is another compatibility issue I found.
Failed to compare desired state to live state: failed to calculate diff from cache: error calculating structured merge diff: error building typed value from live resource: .status.terminatingReplicas: field not declared in schema

[chansuke]

@reggie-k
I found that we use k8s.io.api.core.v1.EventList in three gRPC methods

• ListResourceEvents (server/application/application.proto)
• ListResourceEvents (server/applicationset/applicationset.proto)
• ListEvents ListEvents

my idea is that creating an EventList wrapper message in each proto file to avoid directly returning K8s API types.

message EventList {
  repeated k8s.io.api.core.v1.Event items = 1;
  optional k8s.io.apimachinery.pkg.apis.meta.v1.ListMeta metadata = 2;
}

then update each proto and its implementation. does this approach sound good?

[blakepettersson]

This is a prerequisite for #17694

[dudinea]

Could we use ["k8s.io/apimachinery/pkg/runtime/serializer/protobuf(https://pkg.go.dev/k8s.io/apimachinery@v0.35.0/pkg/runtime/serializer/protobuf)
for kubernetes native objects since we cannot send object directly?

This way we won't have to make a wrapper for any k8s object that we ever want to use.

I think we could make new Methods that would use this library to return results, leave the existing ones as they are for now, document their deprecation and actually remove them when we adopt 1.36.

[reggie-k]

@brantgurga could you elaborate more on the error? Did you install Argo CD on K8s 1.35 / had it manage a K8s 1.35 cluster?
Which version of Argo CD and what was the K8s resource yaml the error was received upon?

[ppapapetrou76]

in the sigs.k8s.io/controller-runtime dep, the Provider.GetEventRecorderFor is also deprecated in favor of Provider.GetEventRecorder, which returns a different EventRecorder interface

[alexymantha]

I have been looking into this for one of our internal controller and I don't think just wrapping it into an extra proto definition like @chansuke showed will work. At least I was not able to make it work, because it will still use the K8s protobuf definition which google.golang.org/protobuf seems unable to serialize properly.

I was looking at what other open-source operators have done and there's a good example from Istio here: https://github.com/istio/istio/pull/58632/changes#diff-8e3960137797989c29531c9c90eba878445a1564df4dd9c1ed9f6f07686b766dL86-R84

they decided to remove the dependency on the k8s definitions directly and just wrap it in google.protobuf.Struct. It works well but you lose some of the type safety and the client needs to cast it to the proper type itself.

[reggie-k]

@chansuke do you think you will have time to work on this?

[chansuke]

@reggie-k
Yes, I can work on this over the weekend and i will take the Istio approach that @alexymantha mentioned.