Faculty of Science wishlist

[miekg]

As per our announcement, we will start to deploy uncloud clusters, but a few extra features in Uncloud would be nice to have.

1. logging: client side #283, so folks can debug their own deployments better
2. Support L4 ingress routing for TCP/UDP ports #108, L4 support. This will allow us to run a DNS server in our "infra/management" clusters. This dns server takes over the role of the default uncl.dev domain, and will prolly see some extra features in the future.
3. Update Corrosion to the latest version from upstream #172, running the latest corrosion. This to prevent a upgrade path in the near future

Nice to have, but can certainly wait:

3. Prometheus metrics, feat: add prometheus #304
4. Cert store for Caddy, Implement distributed storage for Caddy certificates #31

[psviderski]

Thank you for the list!

A few comments on 2.
I'm not yet sure if I'll keep this half-baked feature in uncloud as this brought some confusion in the past. It's not easy to make these records correct in the current distributed model, i.e. when a machine goes down something needs to detect this and update records. Consensus and global view is needed for that. Maybe we just need to make this only a part of the managed offering where we can easily have an authority.

For your use case, if I understand correctly, you aren't going to have too dynamic setups (clusters) so your provisioning workflows can create the required DNS records wherever needed and point them to the machine public IP(s).

Regarding L4 support in Caddy, you should be able to do it already using a custom caddy build and custom x-caddy Caddyfile that defines L4 endpoints and upstreams using Go templates ({{.Upstreams}} mapping) to dynamically update the reload the config when containers change: https://uncloud.run/docs/concepts/ingress/publishing-services#templates

#108 will only make this configuration a bit more concise using x-ports mapping and allow to add/remove L4 proxies with deploying/removing services and without having to update the global x-caddy config manually.

[miekg]

true, we'll do that.

That basically leaves (3), which is not really a blocker either.

[miekg]

ok, so I have my custom stuff. But what is the way to know where my atomdns server runs?

Right now I deployed and looked at the internal IP which I can use in caddy, but some form of automation would be nice here. Maybe labels...?? I.e. caddy understands those and you can use them in the config

[psviderski]

what is the way to know where my atomdns server runs?

Do you mean where atomdns service containers run? Is uc inspect atomdns what you need?

Right now I deployed and looked at the internal IP which I can use in caddy, but some form of automation would be nice here.

Can internal DNS be helpful here? https://uncloud.run/docs/concepts/services/internal-dns
Not sure I understand what you're talking about specifically. Can you please describe what you're trying to achieve?

[miekg]

dumping this here for documentation purposes also

I have a caddy l4 with this config:

{
    layer4 {
        :53 {
            route {
                proxy atomdns.internal:53
            }
        }
    }
}

this loads and runs fine (uc caddy config shows it).

I'm then expecting the following to work:

 dig +tcp @uncloud1.vm.science.ru.nl mx miek.nl                                                                                                      ~cncz/sys/container/caddy main
;; Connection to 131.174.88.77#53(131.174.88.77) for miek.nl failed: connection refused.
;; no servers could be reached
;; Connection to 131.174.88.77#53(131.174.88.77) for miek.nl failed: connection refused.
;; no servers could be reached
;; Connection to 131.174.88.77#53(131.174.88.77) for miek.nl failed: connection refused.
;; no servers could be reached
```

this may be me not understanding caddy l4 well enough

[miekg]

ok, this caddy snippet forward correctly (locally tested):

{
        layer4 {
                udp/:53 {
                        route {
                                proxy udp/atomdns.internal:53
                        }
                }
                tcp/:53 {
                        route {
                                proxy tcp/atomdns.internal:53
                        }
                }
        }
}

but the ports are not opened on the machine, because the caddy controller does not know abou them.
adding a ports doesn't work, and adding them to x-ports also doesn't because then I get an in-use error.

Error: deploy services: start new container: rpc error: code = Internal desc = Error response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint caddy-frvb (4eaeb89f53669fdf1a096b5e7030aff8bcee18749e9c1f2d3e53796c503238ac): failed to bind host port 0.0.0.0:53/tcp: address already in use

and

Error: load compose file(s): service 'caddy': ingress ports in 'x-ports' and 'x-caddy' cannot be specified simultaneously: Caddy config is auto-generated from ingress ports, use only one of them. Host mode ports in 'x-caddy' can be used with 'x-caddy'

and indeed on the machine:

CONTAINER ID   IMAGE                                                COMMAND                  CREATED          STATUS          PORTS                                                                                                                             NAMES
d18fecfb4f00   registry.science.ru.nl/cncz/sys/image/caddy:latest   "caddy run -c /confi…"   12 seconds ago   Up 11 seconds   0.0.0.0:80->80/tcp, [::]:80->80/tcp, 0.0.0.0:443->443/tcp, [::]:443->443/tcp, 0.0.0.0:443->443/udp, [::]:443->443/udp, 2019/tcp   caddy-i2wg

those ports (53 53/udp) are not there.

[miekg]

ah, on different ports it works, it systemd-resolve that blocks the binding

udp        0      0 10.210.0.1:53           0.0.0.0:*                           738/uncloudd
udp        0      0 127.0.0.54:53           0.0.0.0:*                           483/systemd-resolve
udp        0      0 127.0.0.53:53           0.0.0.0:*                           483/systemd-resolve

nope, removing systemd-resolve didn't fix it, but now there is nothing holding the port, but I'm still gettting the error, strange..

update: it is uncloudd holding a 53, but I don't care about that interface.

[miekg]

of course this is what I need:

    x-ports:
      - 80:80@host
      - 443:443@host
      - 443:443/udp@host
      - 131.174.88.77:53:53@host
      - 131.174.88.77:53:53/udp@host
      - 131.174.88.78:53:53@host
      - 131.174.88.78:53:53/udp@host

but this fails because of IP doesn't match. What is needed here is an interface name instead of an IP address, so I can bind to an interface and have it be a global service

[miekg]

#358 is something I really need. Don't care that much about the impl. but I want to be able to forward tcp/udp to a dns server running in the cluster.

[miekg]

boiling down the wishlist to (as said above)

• feat: allow ip prefix in x-ports #358
• Add --output=json to ls subcommands #342
• feat: add prometheus #304

Where the latter 2 are linked together and may be postponed, as this would not really impede our progress. But #358 or equivalent would be nice

[miekg]

pruning (i have minimal pr in flight) and images removal (open pr) is nice as well. Alternative (ssh + docker) must be documented.

[miekg]

I'm testing uncloud plus a few extra PRs merged, via https://github.com/miekg/uncloudplus