Discovery is throttled when there are lots of resources (CRDs)

[seans3]

From k/k: kubernetes/kubernetes#105489

The default client side rate limit is 50qps with a burst of 100.

Kubectl does a discovery pass before doing the operation requested. Discovery involves (some combination of) reading the OpenAPI spec and the "homegrown" discovery information for the purpose of e.g. mapping kinds to resources. Sometimes these can be cached, but even if it is cached, an API call can be required to verify it hasn't changed.

This means that if there are more than 100 group versions, we will hit the client side rate limit in the process of discovery, possibly significantly slowing down the process.

AFAIK there is no way to adjust this limit when using kubectl.

A user report can be found here: kubernetes/kubernetes#101634 (comment)

An easy way to fix this is to disable the client side limit in kubectl. Possibly caching parameters could be adjusted, too. A difficult fix would be to load the minimum amount of discovery documents necessary to perform the task at hand (but some tasks require everything).

[seans3]

/assign @lavalamp
/assign @justinsb
/assign

[seans3]

/triage accepted

[seans3]

@justinsb commented on original bug:

Two contributory things:

I think there's a bug where kubectl get clusterrolebindings.rbac.authorization.k8s.io forces a cache invalidation and full rediscovery, but ``kubectl get clusterrolebindings` does not. The gotcha is that tab completion uses the full name, and of course you're more likely to use tab completion when you have lots of CRDs, making discovery more expensive.

In general, we shouldn't have to do a full discovery to map GVK -> GVR. In the vast majority of cases I think we should be able to just do type discovery in the group and then find the K -> R. (Daniel Smith shared something about Openshift clusters having the G in GVK not equal to the G in GVR, but I would hope we could address that with a fallback). This would be great for controllers, but I don't think it helps for kubectl where we often need shortnames from the apiserver (and in general, kubectl is going to use more of the surface than a controller which only needs GVK -> GVR mapping, so a full discovery is more justified ... ideally we wouldn't invalidate it every time the user typed though!)

[justinsb]

Copied from the original bug:

Two contributory things:

• I think there's a bug where kubectl get clusterrolebindings.rbac.authorization.k8s.io forces a cache invalidation and full rediscovery, but kubectl get clusterrolebindings does not. The gotcha is that tab completion uses the full name, and of course you're more likely to use tab completion when you have lots of CRDs, making discovery more expensive.

• In general, we shouldn't have to do a full discovery to map GVK -> GVR. In the vast majority of cases I think we should be able to just do type discovery in the group and then find the K -> R. (@lavalamp shared something about Openshift clusters having the G in GVK not equal to the G in GVR, but I would hope we could address that with a fallback). This would be great for controllers, but I don't think it helps for kubectl where we often need shortnames from the apiserver (and in general, kubectl is going to use more of the surface than a controller which only needs GVK -> GVR mapping, so a full discovery is more justified ... ideally we wouldn't invalidate it every time the user typed <tab> though!)

[negz]

I just came here to raise this same issue. Over in @crossplane land we'd like to be able to install a lot of CRDs. You can think of Crossplane providers as controller managers that reconcile custom resources representing some set of "external" (to the API server where Crossplane is running) resources - e.g. SQL databases or GCP resources. The long tail of our providers are pretty small (<10 custom resource definitions), but the big ones currently correspond to entire cloud providers (similar to Google KCC) and are thus in the order of 700 custom resource definitions. Hypothetically if someone wanted to install the AWS, GCP, and Azure providers all at the same time that would be around 2,000 CRDs.

We've considered breaking up our providers since it's unlikely folks are actually using even half of (e.g) all 700 AWS APIs but the only real impetus for doing so is avoiding performance bottlenecks. Alongside expensive OpenAPI processing (kubernetes/kubernetes#105932) lengthy discovery processes are the next most annoying consequence of having many CRDs; we're seeing discovery sometimes take minutes.