✨ Add controller_runtime_reconcile_timeouts_total metric to track ReconciliationTimeout timeouts#3382
Merged
k8s-ci-robot merged 1 commit intokubernetes-sigs:mainfrom Nov 8, 2025
Conversation
k8s-ci-robot
added
do-not-merge/work-in-progress
k8s-ci-robot
added
cncf-cla: yes
Contributor
|
Hi @godwinpang. Thanks for your PR. I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/L
godwinpang
changed the title
godwinpang
changed the title
godwinpang
force-pushed
the
add-reconcile-timeouts-metric
branch
2 times, most recently
from
0a03d36 to
39c3abd
Compare
k8s-ci-robot
removed
the
do-not-merge/invalid-commit-message
k8s-ci-robot
removed
the
do-not-merge/work-in-progress
Contributor
Author
|
sorry this ended up requesting a lot of reviewers - It's been a while since I've worked on c-r and I forgot bout the commit conventions hence the force pushes |
Member
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
JoelSpeed
reviewed
Nov 6, 2025
Comment on lines
+168
to
+174
| Expect(func() error { | ||
| Expect(ctrlmetrics.ReconcileTimeouts.WithLabelValues(ctrl.Name).Write(&reconcileTimeouts)).To(Succeed()) | ||
| if reconcileTimeouts.GetCounter().GetValue() != 0.0 { | ||
| return fmt.Errorf("metric reconcile timeouts not reset") | ||
| } | ||
| return nil | ||
| }()).Should(Succeed()) |
Contributor
There was a problem hiding this comment.
Should this not be part of the before each?
Contributor
Author
There was a problem hiding this comment.
Yep good point - fixed and also rewrote to avoid the nesting
This PR adds a new metric controller_runtime_reconcile_timeouts_total to track when a controller's reconciliation has reached a ReconciliationTimeout. This provides visibility into when reconcile operations time out due to the controller-runtime wrapper timeout, allowing users to alert / monitor unexpectedly long running controller reconiliations.
godwinpang
force-pushed
the
add-reconcile-timeouts-metric
branch
from
39c3abd to
93b4145
Compare
Member
|
Thx! /lgtm |
k8s-ci-robot
added
the
lgtm
Contributor
|
LGTM label has been added. DetailsGit tree hash: af34b7fbee2a048e1ad87d730bcc541934bc68bb |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: alvaroaleman, godwinpang The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
timuthy
added a commit
to timuthy/gardener
that referenced
this pull request
Feb 6, 2026
Add `ReconciliationTimeout` panel to the controller details dashboard. The uses the newly introduced `controller_runtime_reconcile_timeouts_total metric` metric (see kubernetes-sigs/controller-runtime#3382).
timuthy
added a commit
to timuthy/gardener
that referenced
this pull request
Feb 6, 2026
Add `ReconciliationTimeout` panel to the controller details dashboard. The uses the newly introduced `controller_runtime_reconcile_timeouts_total metric` metric (see kubernetes-sigs/controller-runtime#3382).
timuthy
added a commit
to timuthy/gardener
that referenced
this pull request
Feb 11, 2026
Add `ReconciliationTimeout` panel to the controller details dashboard. The uses the newly introduced `controller_runtime_reconcile_timeouts_total metric` metric (see kubernetes-sigs/controller-runtime#3382).
gardener-prow bot
pushed a commit
to gardener/gardener
that referenced
this pull request
Feb 13, 2026
…`v0.23.1`, `sigs.k8s.io/controller-tools` to `v0.20.0` (#13982) * `make tidy && make generate` * Solve `OpenTelemetry-Operator` dependency issue The `OpenTelemetry-Operator` includes dependencies to an older version of `sigs.k8s.io/controller-runtime` which is incompatible with the version (v0.23) used in Gardener. Copying the API packages solves the dependency conflict for now. See open-telemetry/opentelemetry-operator#4667 for more information. * Adapt to incompatible changes * Adapt `OpenAPI` generation The generation needs to be adapted due to kubernetes/kube-openapi#563 * Migrate to `k8s.io/client-go/tools/events` Motivated by kubernetes-sigs/controller-runtime#3262 * Refactor: Remove obsolete `rest.Config` constructor * Add `ReconciliationTimeout` panel Add `ReconciliationTimeout` panel to the controller details dashboard. The uses the newly introduced `controller_runtime_reconcile_timeouts_total metric` metric (see kubernetes-sigs/controller-runtime#3382). * Change `NodeAgentAuthorizer` integration test Don't use the event recorder to test patch permissions. We've seen cases where the event recorder drops events if they are recorded too quickly.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add controller_runtime_reconcile_timeouts_total metric
Summary
This PR adds a new metric
controller_runtime_reconcile_timeouts_totalto track when a controller's reconciliation has reached aReconciliationTimeout. This provides visibility into when reconcile operations time out due to the controller-runtime wrapper timeout, allowing users to alert / monitor unexpectedly long running controller reconiliations.Problem
The
ReconciliationTimeoutfeature was added to prevent head-of-line (HOL) blocking by ensuring that a single reconcile cannot block a worker indefinitely. However, when a reconcile exits becauseReconciliationTimeoutfired, the existing metrics treat it like a normal return. There was no way to distinguish between:ReconciliationTimeoutcontext timeoutMetric
controller(controller name)