OIDC config via secret fails intermittently

[dfry]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When the OIDC configuration is provided with reference to a secret value, some times the secret is interpreted correctly, other times it is not, resulting in sending the actual string ($argo-oidc:clientid) to the OIDC provider instead of the value from the secret. When this happens, a delete of the argocd server pod will resolve the issue.

To Reproduce

Deploy argo with the below oidc config map elements. After a random period of time, even after a initial successful OIDC login has been performed, when the OIDC token refresh occurs, it will fail with the below warning messages in the logs.

apiVersion: v1
kind: Secret
metadata:
  name: argo-oidc
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd
data:
  clientid:  redacted
  clientsecret: redacted
type: Opaque
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  application.instanceLabelKey: argocd.argoproj.io/instance
  kustomize.buildOptions: '--enable-helm'
  oidc.config: |
    name: Gitlab
    issuer: redacted
    clientID: $argo-oidc:clientid
    clientSecret: $argo-oidc:clientsecret
    requestedScopes: ["openid", "profile", "email", "read_api"]

Expected behavior

Secret references should always be resolved and not treated as plain text.

Screenshots

Version

Argo CD: v2.9.7+fbb6b20
Build Date: 2024-03-01T22:28:14Z
Go Version: go1.21.3
Go Compiler: gc
Platform: linux/amd64
jsonnet: v0.20.0
kustomize: v5.2.1 2023-10-19T20:13:51Z
Helm: v3.13.2+g2a2fb3b
kubectl: v0.24.17

Logs

time="2024-05-15T12:26:15Z" level=warning msg="config referenced '$argo-oidc:clientid', but key does not exist in secret"
time="2024-05-15T12:26:15Z" level=warning msg="config referenced '$argo-oidc:clientsecret', but key does not exist in secret"

[cc-robertson]

Same problem for us! We reproduced this issue, it happens quite often

[agaudreault]

Is there a process updating this secrets? This error can happen if you have an external system updating the secrets and potentially removing the annotation app.kubernetes.io/part-of: argocd.

If you can provide any logs related to the secrets lister, that would be helpful. Argo will use Kubernetes primitives to list all the secrets with that annotation in the argo namespace.

[dfry]

Yep, the secret is generated by external secrets. The only relevant logs have been posted already.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: argo-oidc
spec:
  refreshInterval: 1h

  secretStoreRef:
    kind: ClusterSecretStore
    name: tenant-vault-secret-store

  target:
    name: argo-oidc
    creationPolicy: Owner
    template:
      metadata:
        labels:
          app.kubernetes.io/part-of: argocd
  data:
    - secretKey: clientid
      remoteRef: 
        key: {{ cluster_name }}/argocd_oauth_client_id
        property: value
    - secretKey: clientsecret
      remoteRef: 
        key: {{ cluster_name }}/argocd_oauth_client_secret
        property: value

[agaudreault]

@dfry What is the Kubernetes version on which Argo is deployed? 🤔

[dfry]

1.29.4

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and tell us if the issue is still present, please?

[laurivosandi]

I can confirm clientID expansion doesn't work reliably on v3.1.9+8665140 either:

clientID: $oidc-client-argocd-owner-secrets:OIDC_CLIENT_ID
cliClientID: $oidc-client-argocd-owner-secrets:OIDC_CLIENT_ID
clientSecret: $oidc-client-argocd-owner-secrets:OIDC_CLIENT_SECRET

[leonardfactory]

Edit: My bad, I was using annotations instead of labels. Seems to be working correctly with dex.config (didn't try with oidc.config). I'm leaving this message in case it may help someone else.

Reproduced on v3.1.9+8665140 too:

level=warning msg="config referenced '$argocd-custom-secret:googleOidc.clientSecret', but key does not exist in secret"

dex.config: |
  connectors:
  - type: google
    config:
      clientSecret: $argocd-custom-secret:googleOidc.clientSecret

• Secret is generated by Sealed Secrets like this:

apiVersion: v1
data:
  googleOidc.clientSecret: ++++++++
kind: Secret
metadata:
  name: argocd-custom-secret
  namespace: argocd
  annotations: # <- this was my issues, should have been labels !
    app.kubernetes.io/name: argocd-custom-secret
    app.kubernetes.io/part-of: argocd
type: Opaque
# ...