Pipeline controller deletes PipelineRuns when two cluster contexts share the same cluster

[derryos]

By default, Prow uses the default cluster

We wanted to use multiple build clusters (for various workflows/testing upgrades) so we created cluster-a/cluster-b/cluster-c as well as leaving the 'default'. As we wanted to change the default cluster we use post upgrade, we used specific named clusters as the cluster: variable within our prow pre/post/periodic jobs instead of leaving unnamed (default) but now we have an issue where prow is getting confused between 2 clusters (default and another) with the same kubeconfig.

This is the code that is triggering:
https://github.com/kubernetes-sigs/prow/blob/main/cmd/pipeline/controller.go#L463

Essentially the same cluster context is duplicated with different names and the prowjobs are being deleted as the default cluster and the other cluster are technically the same build cluster, just named differently.

Is there a way to deal with this easily or is code needed to validate that the contexts are for the same cluster but named differently?

[petr-muller]

@derryos so I understand this is happening only for the tekton-backed Prowjobs (handled by pipeline controller) for you, not plain-old pod-backed ones? I have not heard about similar issues in the big multi-cluster Prow instances I'm familiar with (k8s and openshift) but neither is using pipeline-backed jobs, so if the confusion you describe (that tbh I'm not really clear about from the report but that may be just me not knowing the pipeline controller well enough) happens only in the pipeline-controller it just sounds like the controller is buggy (I guess most work on the named/multi build farm support was done with the plain old pod-bakced prowjobs in mind, true)

[petr-muller]

/retitle Pipeline controller deletes PipelineRuns when two cluster contexts share the same cluster

The root cause is in how the pipeline controller sets up per-cluster informers. When multiple kubeconfig contexts point to the same underlying cluster, each context gets its own independent informer watching the same resources. When a PipelineRun event arrives via the "wrong" context's informer, the reconciliation check at cmd/pipeline/controller.go:463 (ClusterToCtx(pj.Spec.Cluster) != ctx) evaluates to true, setting wantPipelineRun = false and causing the controller to delete the PipelineRun. This only affects the pipeline (Tekton) controller; other Prow controllers operate in single-cluster mode and are not impacted.

The fix would be to add startup validation in cmd/pipeline/main.go that detects when multiple kubeconfig contexts resolve to the same cluster API server endpoint. There's already precedent for this kind of deduplication in the same file: the InClusterContext → DefaultClusterAlias mapping at lines 127-136 handles the specific case of empty-string vs "default" aliasing. The general case (any two contexts sharing a server URL) can follow the same pattern. As a workaround until this is fixed, ensure each kubeconfig context name maps to a distinct cluster — remove the duplicate context or avoid configuring cluster: in ProwJobs to use a named alias that shares a cluster with default.

/kind bug
/help-wanted

Triage information

This comment was made by experimental Claude triage helper. I reviewed the content and I hope it is useful and not AI slop. If you have feedback please reach out to me.

Full triage: https://github.com/petr-muller/prow/blob/issue-triage-609/ISSUE-TRIAGE.md