Implement distributed storage for Caddy certificates

[psviderski]

When using multiple machines with Caddy behind a TCP load balancer or a DNS record with multiple IPs, certificates can take a while to issue. The acme challenge bounces around to different machines until it hits the correct one. Moreover, if Cloudflare proxy record is used with multiple addresses and one instance is able to successfully issue a certificate, Cloudflare may not try to send requests for ACME challenge to other instances or send it very rarely. In practice, I saw the cases when instances failed to issue certificates for days or weeks.

Possible solutions:

• Share the certificate storage (file system) between caddy containers
• Consider using alternative storage module for certificates or implement our own, e.g. backed by Corrosion: https://caddyserver.com/docs/json/storage/

As the first step for implementing a distributed storage, we can fork the default file_system storage that uses the certmagic's one and only implement Store and Load methods for keys that store challenge tokens. We can store challenge tokens in Corrosion that will share them among all Caddy instances.

We can perhaps start even without a distributed lock. In the case when multiple instances start issuing a certificate for the same domain at the same time, they may override each other's token but after a few retries they all should be able to succeed. We can try adding optimistic locking by failing if a record for the key exists and ignore it after a reasonable timeout.

[MCCC4]

Just hit a similar problem setting up split-horizon DNS

As a workaround it's possible to set up a DNS challenge, there are multiple steps involved but it's doable:

• Your DNS registrar must be supported by one of the Caddy's DNS modules - https://github.com/caddy-dns - or you have to make do with this: https://caddyserver.com/docs/caddyfile/directives/tls#dns_challenge_override_domain
• Build a Caddy docker image that includes the correct DNS module (--with <module>):

FROM golang:1.25-alpine AS builder

RUN go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
RUN ./bin/xcaddy build v2.10.2 --with github.com/caddy-dns/ovh

FROM caddy:2.10.2-alpine
COPY --from=builder /go/caddy /usr/bin/caddy

• Distribute the built image across your machines
• Create a custom Caddyfile with the TLS challenge configured:

*.<your-domain> {
  tls {
    dns <module-name> {
      ... config - see the README of the module you need ...
    }
    resolvers 8.8.8.8 1.1.1.1  # <- Needed this for split-horizon DNS
  }
}

• Redeploy Caddy with the image and config:

uc caddy deploy --caddyfile <your-caddyfile> --image <your-custom-image>

[psviderski]

We can try to implement a dirty first iteration by just syncing the challenge files in the caddy data dir between machines via the shared Corrosion cluster table (key/value). We can do this outside Caddy in the uncloudd daemon so there is no need to build a Caddy image with our custom module.

[miekg]

For #108 a custom caddy might be needed anyway and having Caddy being able to talk Corrosion might help with other use cases as well

[psviderski]

Yeah that's true. I think it will make sense to expose an API on top of Corrosion to give access to watch for cluster changes and selectively change something. Corrosion is too low level and too permissive for external integrations. It doesn't have a layer for controlling permissions.

[miekg]

I think https://letsencrypt.org/2026/02/18/dns-persist-01 will mean multiple caddy's can just get the certs - haven't doubled checked though as LE hasn't deployed this yet

[miekg]

I've done numerous things with caddy (well caddy v1, including forking it into coredns), and caddy v2 is even nicer (code wise). Would an uncloud caddy plugin that talks with corrosion help? That mostly lifting the corrosion code from uncloud (or re-using!) and creating a caddy (FileSystem) module, uncaddy or whatever it needs to be called.

[miekg]

this PR (in my forked repo) implement the certstorage interface miekg#1

So the plan I have is this:

• in uncloud we add a pkg/caddy to uses internal/machine/store, this is so that we do not have to change anything nd really control what we expor
• then in a second repo, named caddy or uncaddy we only import pkg/caddy in a tiny Go file, with go.mod and friends. This is only needed because otherwise it is not a plugn caddy can use. It can't (IIRC) to pkg/caddy from uncloud directory
• then using the above repo and the l4 caddy plugin repo we build a custom caddy - unsure where this code shoud live, probably in the main uncloud repo (Makefile?).

update

I think I have most the pieces, it's bigger than I would like though. Also needed a bunch of changes to add the corrosion endpoint to the caddy config

[miekg]

If this at some point is merged, is there is also need to implement caddy.FileSystem? That supports serving files from corrosion?

[psviderski]

If I understand miekg#1 correctly, it proposes to use Corrosion as the main Caddy storage for everything, including the certificates as well. Here are a few concerns:

• So far we didn't store any sensitive data in Corrosion. Well, almost, we store a token for updating the uncloud-dns records. But I'm still considering it an experimental feature so I made a shortcut.

  If we want to store certificates there, we need to start treating it more seriously. For example, make sure the auth is set up: Enable auth for local requests to Corrosion API #110 and maybe use some basic encryption at rest so that when the corrosion db is backed up, it doesn't contain private keys in plain text. The store.db on the file system has 0644 permissions by default so need to constraint it further. Although, the /var/lib/uncloud/corrosion dir is 0700 which shouldn't allow access to nested files to non-uncloud/root users.

  Maybe not right away and we can iterate on this but this is something to keep in mind.

• We need to set the storage global option to use our own storage. For this we would need to get into the Caddyfile parsing/assembling zone to be able to inject an option in an existing global block. Caddyfile can have only one global block. And the user can define it in their custom Caddyfile: https://uncloud.run/docs/concepts/ingress/managing-caddy#deploying-or-updating-caddy.

  Alternatively, we can try to not mangle the Caddyfile but instead patch the adapted JSON config we load into the Caddy admin API. This should be much easier to do but this would be implicit and not visible when getting the config with uc caddy config. Not a big deal most likely.

So overall using Corrosion as a general purpose storage (of course if its eventual consistency is acceptable) like etcd in k8s is a powerful approach. The only concern I have before starting doing it is figuring out the security model for it and whether we're happy with the fact that any machine in the cluster can read and write any data in the store.

With etcd, its scope is at least limited to control-plane nodes, not all. But we have a control-planeless setup so every machine runs a Corrosion replica with full write permissions. It's a bit tangential, but the gRPC API layer has a similar permissive model at the moment. But we can introduce tokens/roles/scopes later if we want to harden it.

I believe this is not something we would be able to do with Corrosion. All nodes are equally trusted in its model.

That's why my proposal here was to try to use Corrosion only for the temporary challenge tokens but still store certificates on the file system. Or maybe even use gRPC API to broadcast storage requests or have an alternative approach to sync challenges/certs between machines like rsync. I've heard that someone successfully used https://github.com/lsyncd/lsyncd running on each machine to sync the directories with caddy certificates with other machines.

Might be hard to build the complete mental model of how things work and what the tradeoffs are in a short period of time but would like to hear your thoughts.

[miekg]

yes, good points. Regarding those:

• We should def. change those permissions to be more strict. I would say the certificate keys is not exactly the data that I would consider sensitive, but your point stands. I'm just saying maybe we can still say we don't care. About uncloud-dns in my implementation that returned domain name is the token (using a slightly longer random name), so that's already semi-public.

• my branch injects a global section, but doesn't merge it with x-caddy, i'll add that too.

I.e. I think I'm saying the following:

1. Do the minimal thing to make things more secure on the machines
2. Implement caddy Storage
3. Maybe also implement caddy FileSystem, but this is less important
4. Think about corrosion/security later ... (like the sql interface of corrossion, grown tired of etcd :) )

[miekg]

ugh, the x-caddy makes it somewhat complex, and very if-then-else heave; not elegant at least...

If caddy can discover the corrosion endpoint when starting it may be possible to register a Storage automatically with resorting to printing something in the Caddyfile

[tonyo]

A couple thoughts on the security model:

Corrosion as an underlying layer drastically limits what we can guarantee security-wise (any machine can read and update the state), and that's the reality we have to live with as long as Corrosion is in use. It works quite well for the current needs from what I see and has successfully helped to validate the overall concept of Uncloud in the early days, but I think we can all agree that it's not the ultimate technology that must power Uncloud for all eternity.

Thinking out loud about more secure and auditable solutions for the future, it'd be sweet to have a communication layer that supports authenticated/signed, append-only, and scoped operations.
For example: an authenticated admin CLI operator could perform any operations in the cluster, they will be verified and accepted by the cluster nodes. Less privileged operators can perform a subset of operations on a subset of services. Services and nodes also have identities and are normally only allowed to do the bare minimum; if a node is compromised, its uncloud daemon cannot for example "order" the cluster to deploy a new service or delete an existing one, or read all configs/secrets that were not present on the node at the time.
Applying to the Caddy/certificates discussion -- Caddy (and node itself) would have access to writing and reading the certificate tokens/keys, but not accessing data for other services; as in, the compromised node would try to issue an operation for that, but the rest of the cluster would not verify and accept it.
A brief research yielded that NATS might already have a bunch of reusable primitives that can be used to implement this (zero-trust, distributed) auth model: https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/jwt
I remember @psviderski you were somewhat skeptical about NATS ;) But I'd be curious to explore this approach a bit further at some point, a nice eventual bonus being that we can embed the useful NATS parts into Uncloud itself and get rid of the Corrosion daemon as a separate component.

Regarding the current issue, my suggestion would be:

1. Implement the low-hanging security fixes for the current architecture (such as file permissions)
2. Explain and highlight the current security model in the docs more prominently, to avoid any misunderstanding on what happens e.g. when one node in the cluster gets compromised
3. Move forward with storing the temp challenge tokens in Corrosion
4. Start discussing the successor for Corrosion in the foreseeable future, that'd eventually become the foundation of Uncloud 2.0.