argocd-server services randomly new / old / expired cerificate from secret argocd-server-tls

[vx-github]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.

Describe the bug

Since version 2.4.3 I noticed argocd-server is sometimes serving older / expired certs that should not be served any more. We first noticed this after the upgrade from 2.2.5 to 2.4.11. The problem exists in 2.4.12 too.

To Reproduce
Use this https://github.com/vx-github/vx-argocd-cert-bug to easily reproduce the issue locally in a kind cluster.

Expected behavior

Expected behavior would be that argcd-server only serves / uses the cert in argocd-server-tls secret if it exists and doesn't serves / uses certs it was not supposed to (old / expired ones).

[jessesuen]

Is this a problem that clears itself after a restart of argocd-server pods? I feel the randomly generated certs should no longer be generated so long as argocd-server-tls is populated with explicit ones.

[vx-github]

@jessesuen
Yes a pod restart fixes if for a while but the problems returns. In our original setup we use cert-manager and after the upgrade to 2.4.11 we saw Argo CD serving expired (old) certificates. I narrowed the problem down to argocd-server by leaving out cert-manager. So it is not about Argo CD creating new selfsigned certs, but using old ones instead. Does argocd-server cache these certificates somewhere?

My observation is that certificates that were once served (since last pod start) by argocd-server (any: Argo CD selfsigned, certs from cert-manager or certs that where manually placed in secret argocd-server-tls) and were replaced (by updating the secret argocd-server-tls) do show up again after some time. With the above steps to reproduce the problem, it is in matter of tens of seconds that the old Argo CD selfsigned cert shows up again (same expiry date/time). If you would update argocd-serer-tls again (new cert), then you have two old certs that will show up randomly (just let the loop run long enough).

[vx-github]

@jessesuen
So maybe to make it more visual. We have some monitors in Elastic that show the expiry dates of the certs that is on the Argo CD endpoint.

For some environments we still run Argo CD v2.4.11 and the below a screenshot shows argocd-server returning an older (previous) cert some times:

We don't see this happening in Argo CD 2.4.2 (and this is the expected behavior):

[frenkyj]

Hello, I can confirm that we see the same behaviour in our env. Running v2.4.9. thanks for your help

[cathelijne]

Reproduced this in a k3d cluster. Argocd versions 2.4.[8-12] (didn't have time to go further back). Tried with self-signed certs as described above, and also self-signed, Hashicorp Vault and acme provider from cert-manager. Certs get cached somehow, it seems, even though the secret in the cluster has the correct (updated) certs.

in short: kubectl get secret gives me the correct cert, openssl s_client gives me a random one. It doesn't matter where and how the cert was generated.

[wmgroot]

We saw this today running v2.4.7+81630e6.
A restart of the argo-server pods resolved the issue for now.

[vx-github]

@jessesuen
The problem still exists in v2.5.2 and in v2.4.17.
I created a repository with some steps to easily reproduce the problem locally: https://github.com/vx-github/vx-argocd-cert-bug

[vx-github]

@crenshaw-dev
Please have a look at this.
(https://cloud-native.slack.com/archives/C01TSERG0KZ/p1664225432932189)

[vx-github]

@jessesuen @crenshaw-dev
Problem still exists in v2.4.18 and v2.5.5. It also exist in the RC: v2.6.0-rc1

Please try to reproduce: https://github.com/vx-github/vx-argocd-cert-bug

[vx-github]

@jessesuen @crenshaw-dev
Problem still exists in v2.4.19 and v2.5.7. It also exist in the RC: v2.6.0-rc4

Please try to reproduce: https://github.com/vx-github/vx-argocd-cert-bug

[blakepettersson]

I can repro this, looking at the logs I see this:

[argocd-application-controller] time="2023-01-20T19:45:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:45:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000a15980]"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000930360]"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="tls certificate modified. restarting"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="shutting down settings watch"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="Shut down requested"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="0xc000930360 unsubscribed from settings updates"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="rbac configmap informer cancelled"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="argocd v2.6.0+cb2db73.dirty serving on port 8080 (url: , tls: true, namespace: argocd, sso: false)"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="Enabled application namespace patterns: argocd"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="0xc000dbb1a0 subscribed to settings updates"
[argocd-applicationset-controller] time="2023-01-20T19:45:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="Starting rbac config informer"
[dex] time="2023-01-20T19:45:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[dex] time="2023-01-20T19:45:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000349b00]"
[dex] time="2023-01-20T19:45:48Z" level=info msg="dex config unmodified"
[argocd-server] time="2023-01-20T19:45:48Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
[argocd-application-controller] time="2023-01-20T19:45:56Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-repo-server] time="2023-01-20T19:46:04Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-
01-20T19:46:04Z" grpc.time_ms=0.018 span.kind=server system=grpc
[argocd-server] time="2023-01-20T19:46:05Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:06Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:16Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:26Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-repo-server] time="2023-01-20T19:46:34Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=Check grpc.service=grpc.health.v1.Health grpc.start_time="2023-
01-20T19:46:34Z" grpc.time_ms=0.018 span.kind=server system=grpc
[argocd-server] time="2023-01-20T19:46:35Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:36Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:46Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-application-controller] time="2023-01-20T19:46:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000dbb1a0]"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="tls certificate modified. restarting"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="shutting down settings watch"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="Shut down requested"
[dex] time="2023-01-20T19:46:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="0xc000dbb1a0 unsubscribed from settings updates"
[dex] time="2023-01-20T19:46:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000349b00]"
[argocd-application-controller] time="2023-01-20T19:46:48Z" level=info msg="Notifying 1 settings subscribers: [0xc000a15980]"
[dex] time="2023-01-20T19:46:48Z" level=info msg="dex config unmodified"
[argocd-server] time="2023-01-20T19:46:48Z" level=info msg="rbac configmap informer cancelled"
[argocd-applicationset-controller] time="2023-01-20T19:46:48Z" level=info msg="Loading TLS configuration from secret argocd/argocd-server-tls"

I guess the question I have is why a single update to the secret invokes updateSettingsFromSecret multiple times 🤔

[blakepettersson]

With these changes I can make @vx-github's test harness work as expected

In server.go, change the TLS config to use GetCertificate

		tlsConfig := tls.Config{}
		if a.TLSConfigCustomizer != nil {
			a.TLSConfigCustomizer(&tlsConfig)
		}
		tlsConfig.GetCertificate = func(info *tls.ClientHelloInfo) (*tls.Certificate, error) {
			getSettings, err := a.settingsMgr.GetSettings()
			if err != nil {
				return nil, err
			}
			return getSettings.Certificate, nil
		}

as well as by removing the shutdown logic in watchSettings. We'd still want to keep that part, so this would need a bit of a think on how to maintain that but to otherwise use GetCertificate for the TLS settings.