Scale testing

[sbueringer]

This issue provides an overview over ongoing and upcoming tasks around scale testing.

Scale:

• PriorityQueue

  • [Reconcile latency] ✨ Enable PriorityQueue per default #13171

• ReconcilerRateLimiting:

  • [Memory] ✨ Promote ReconcileRateLimiting to beta (enabled per default) #13373
  • [Memory] 🌱 Increase MHC rate limiting and filter events #13422
  • [Memory] 🐛 Fix exponential backoff with ReconcilerRateLimiting #13416

• Optimize cache configuration

  • [Memory] ✨ Optimize cache configuration of CABPK & standardize cache/client setup #13407
  • [Memory] ✨ Optimize cache configuration of KCP #13460
  • [Memory] ✨ Optimize cache configuration of core CAPI #13488

• Admission webhook improvements:

  • [Memory] CR: ✨ Reduce memory usage of default webhooks controller-runtime#3463

• Conversion webhook improvements:

  • [Memory] conversion/convert gets called a lot for unserved versions (even though we never use old apiVersions) Skip creating storages for non-stored and non-served versions kubernetes/kubernetes#130704
  • [Memory] 🌱 Introduce conversion.MarshalDataUnsafeNoCopy to avoid unnecessary memory allocations during conversion #13402

• Patch helper improvements:

  • [Memory] 🌱 Stop using patch helper when enforcing finalizers #13365
  • [Memory] 🌱 Improve patch helper for no-op cases #13364
  • [Memory] 🌱 Optimise patch calls #13367
  • [Reduce apiserver calls] 🌱 Add safeguard to patchHelper to avoid sending empty patches to the apiserver #13403

• Misc:

  • [Reduce apiserver calls] 🌱 Avoid unsetting nodeDeletionTimeoutSeconds during Machine deletion #13401
  • [Reduce apiserver calls] 🌱 KCP should read only KCP machines #13457
  • [Reduce apiserver calls] 🌱 Remove live list Machine call in KCP #13458
  • [Reduce apiserver calls] ✨ Refresh cache entries on cache hit in SSA caches #13459
  • ✨ Tune controller concurrency and cache timeout for scale #13496

• Reduce log verbosity:

  • 🌱 Remove stack traces from ClusterCache errors #13396
  • 🌱 Drop or reduce verbosity for noisy logs #13420
  • 🌱 Use klog.Obj in log messages #13421

Scale testing:

• 🌱 Enable native histograms in Grafana / Prometheus #13304
• 🌱 Inmemory APIserver fails for unsupported fieldSelectors #13306
• ✨ Improve scale test setup: add infra CRDs to dashboards, improve/fix dashboards, enable native histograms #13354
• 🐛 Fix goroutine & memory leak in inmemory provider #13361
• 🌱 Revive the debug endpoint for CAPDev in-memory #13423
• 🌱 Improve scale test for re-entrancy and improve observability stack #13492
• Improve hot restarts in CAPDev
  • See changes to test/infrastructure/vcsim/controllers in ✨ Improve scale testing machinery cluster-api-provider-vsphere#3878

Potential follow-ups:

• Consider introducing a direct link between syncPeriod and SSA cache TTL: ✨ Refresh cache entries on cache hit in SSA caches #13459 (comment)
• Consider caching earlier in dryrun.go: ✨ Refresh cache entries on cache hit in SSA caches #13459 (comment)