Can Kilo Work With Nodes Behind NAT?

[gravypod]

I've been looking into using Kubernetes in an at-edge setting. In this type of deployment I'd be setting up nodes behind other people's NAT'ed networks. Kubernete's API and CRDs make a lot of things I need to do very easy to accomplish (daemonsets, service meshes, and config management, etc) very simple. Wireguard would provide a transparent security layer. In my application I don't mind the high latency with communications to the API server and other things. One thing that I don't control in my deployment is the router at the locations of deployment. I can guarantee there will be a network connection that will be able to speak to my api server but I cannot forward ports.

I noticed in your documentation you must provide at least one public IP to each region. Is there some way to use kilo to avoid this constraint? What does this constraint come from? Is this some inherent feature of WG?

[squat]

Hey @gravypod at least one peer in the WireGuard mesh will need to have a public IP and a publicly accessible port, eg if you run all nodes on the edge behind NAT but run the control plane in the cloud. At first, the node in the cloud will not be able to connect to the edge nodes, but once the edge nodes are booted and connect to the cloud node, the IP and port of the edge node will automatically be recorded in the cloud node’s configuration. In particular, you can use a helpful WireGuard configuration feature called PersistentKeepAlive, which instructs WireGuard to periodically send a keep alive packet to keep the port open and I’m this way be able to traverse NAT gateways. In this case WireGuard will work just fine: a node in the cloud has a publicly accessible IP and port and all nodes behind users’ NAT can connect to it. I use this exact configuration in several deployments and it is one of the primary use cases for this project :)

[frbncis]

Do you have any documentation on how to apply the PersistentKeepAlive setting on the edge nodes? I've tried editing /var/lib/kilo/conf but that gets overwritten and I don't see any annotations available to specify the value on a given node.

Many thanks for this project!

[squat]

Hi @frbncis right now the PersistentKeepalive setting can be only be adjusted for Peers added via the Peer custom resource. Currently it cannot be set for nodes that are directly part of the Kubernetes cluster. It would not be difficult to add at all. It could be implemented via another node annotation. Would that be helpful? Let me know and id be happy to write the feature (or review a PR :))

[frbncis]

Out of curiosity, I decided to give a try at implementing here: master...frbncis:node-keepalives

Seems to be working for me, on the node behind the NAT:

interface: kilo0
  public key: L769oKuo5GItkx3hC4oDrfFrHaNf/fF+7NDEJLWfFyQ=
  private key: (hidden)
  listening port: 51820

peer: KIp5hMoLTayQ9XndUvO4ZkBC9d65jLZoGcMWTg+0jGo=
  endpoint: <cloud-node-public-ip>:51820
  allowed ips: 10.42.0.0/24, 10.20.0.5/32, 10.4.0.1/32
  latest handshake: 21 seconds ago
  transfer: 83.57 KiB received, 253.40 KiB sent
  persistent keepalive: every 2 seconds

The cloud node is annotated with kilo.squat.ai/wireguard-persistent-keepalive: 2

If I'm on the right path, I'd be happy iterate on it and put it in as a PR.

[squat]

💯 totally well done! I read through the commit and the only comment I would have is to keep the annotation and struct field name succinct: WireGuardPersistentKeepalive -> PersistentKeepalive

Nicely done :)

[squat]

Tht would be a great feature to document in a short doc. We can add that in a follow up PR of course

[squat]

Fixed by #31 (thanks @frbncis)

Please reopen if you see this doesn't address your use-case

[carlosrmendes]

Hi,
I have one master node in region A with a public ip and a worker node in region B behind a NAT (two separate networks).

After deploy Kilo I annotated both nodes to force external ip (master with own public ip and worker with NAT public ip) and to set the related location on each (master: region-a, worker: region-b).

Checking the wireguard peers in the master, with wg command, I can see the peer of the worker, with the NAT public ip as endpoint, but the port is different than the wireguard listen port set on the worker node.

I can also see that the an handshake was made successfully, but after 30s approximately, the Kilo recreate the peer because it detects differences on configuration (log: 'WireGuard configurations are different'), due to the endpoint port and interrupting existing connections.

How can I solve this?
Thanks in advance.

[squat]

Hi @carlosrmendes let's discuss this in a new issue :) please open a new issue and post the WireGuard configurations for the master and worker so we can see what is going on (we can redact keys and IPs of course).

Thanks,
-Lucas

[eddiewang]

Hi there @carlosrmendes and @squat - were you two ever able to trace this down? I am also seeing an issue where the Wireguard connection is made, but Kilo is overwriting the Endpoint in the config, causing the connection to reset, and the host to be unreachable.

In the Kilo pods, logs read that WireGuard configurations are different.

Is there any way to statically set the static Endpoint in the peer yaml?

[squat]

Hi @eddiewang yes this bug was fixed a few months ago. Your issue sounds a bit different actually. Can you file a new issue and describe your scenario in detail?

[eddiewang]

@squat turns out it has to do with kilo overwriting the endpoint once the connection was established. I manually specified the endpoint in the yaml and things persisted again. although this is running rke with flannel as the primary CNI. I can't seem to get kilo to behave well in CNI mode.