Allow nodes/peers behind NAT to communicate directly (NAT to NAT)

[stv0g]

I am looking into a scenario having multiple cluster segment leaders behind NAT routers.

Usually a direct communication between these nodes is not possible as the local endpoint of these nodes gets mangled by NAT.

Jordan Whited has written a pretty nice writeup of a UDP hole punch technique for Wireguard to circumvent this issue:
https://www.jordanwhited.com/posts/wireguard-endpoint-discovery-nat-traversal/

His solution is both elegant and simple to implement.
Its based on the idea to use a public node as a registry which records the public facing endpoints of the NATed nodes an distributing them to all other nodes.

kilo has support for roaming of NATed nodes by recognising endpoint updates without overwriting them during the reconciliation phase (see #26, #34, mesh.updateNATEndpoints()).

I am proposing that we go a step further and also distribute these updated endpoints to other nodes in the cluster.
This could be done by letting kilo updating the kilo.squat.ai/force-endpoint annotation of these nodes.
This would replace the role of Jordan's CoreDNS plugin as we could use the Kubernetes API server.

However, I think it might be more elegant to introduce a new annotation kilo.squat.ai/endpoint to differentiate between a forced endpoint by the user and a dynamically recognized one by kilo.

What are your opinions on this proposal? I would give this a try if it is welcomed :)

[stv0g]

Okay, I digged a bit deeper into Kilo's codebase. My findings so far:

• Kilo already uses an annotation for the real endpoint. Its named kilo.squat.ai/endpoint
• Manual updates of this annotation are automatically synced to all Wireguard nodes as by the usual reconciliation process.
• The periodic reconciliation timer triggers also a call to mesh.checkIn() which in turn should update the kilo.squat.ai/endpoint annotation with the real values learned by mesh.updateNATEndpoints() (which itself is called by mesh.applyTopology() which is also called by the periodic reconciliation timer).

So, to summarize, it appears to me that Kilo already implements my proposed change 😄

@squat Could you confirm this?

[sergeimonakhov]

@stv0g hey! you right it implemented, but need have one public ip on node or forwarded port in nat

[stv0g]

@squat mentioned in #102 that it is currently not implemented. I will need to do some further testing to confirm this.

Just by looking at the code it seems that kilo already implements the approach which Jordan Whited described in his blog post (see above).

but need have one public ip on node or forwarded port in nat

@D1abloRUS Can you elaborate on that? One public IP of the communications partners? Or one public IP just for signalling the NATed endpoints?

[sergeimonakhov]

@stv0g I was in a little hurry, sorry. I have the following topology and there are problems with the connection between two nodes located behind nat between themselves. From a public node, both nodes are available

[mohsinhijazee]

This is really an interesting prospect. Just wondering how tailscale is doing it via STUN+DERP so maybe there could be a possibility to run coturn on one of the clusters (or any other cheaper public cloud instanceas) and then integrate it with kilo? As far as I have seen, tailscale also is using wireguard-go but I am not sure how and at what point they mixin the whole NAT traversal.

[squat]

@stv0g thanks for starting this conversation! NAT-NAT communication had been on the wishlist for Kilo for a while and it makes me happy to see other people excited about it as well. Your description pretty much sums up the current state of the project:

• Kilo allows peers to roam even if they are behind NAT, thanks to the updateNATEndpoints goroutine
• Kilo does not currently allow logical locations to have leaders behind NAT (without port forwarding); all location leaders must have self-discovered or user-set public endpoints
• Kilo does not enable two peers behind NAT to establish a connection
• Once a peer behind NAT has established a connection with a node, Kilo does not enable nodes in other locations to communicate with that peer unless it expressly establishes a connection with that node

I would love to see Kilo support the following functionality:

1. Allow an entire logical location to be behind NAT, ie the leader has no public endpoint but it can establish a connection with the locations that do have public endpoints
2. Enable endpoint discovery, e.g once a peer or node behind NAT establishes a connection with a node, the destination node announces the discovered publicly accessible endpoint, e.g. via a kilo.squat.ai/discovered-endpoint annotation on the Peer CR or Node resource, which is used as the endpoint in WireGuard configs generated by Kilo, enabling other nodes and peers behind NAT to connect to the endpoint

I agree that it will probably be less confusing to put the discovered endpoint in a new annotation, at least to start. I would be more than happy to review any PRs for these features 🎉

[squat]

Solving feature 1 should essentially be a question of relaxing the conditions that Kilo checks when it considers if a node is "ready". Currently, Kilo requires that nodes have an endpoint, as it was a requirement that at least one node in each location have a public endpoint. By removing this requirement, we allow all the nodes in a location to be fully behind NAT and to not know their own publicly accessible endpoint. This paves the way for feature 2, which is nodes announcing the discovered endpoints of NATed nodes.

[JulienVdG]

Hello, I think feature 1 is already there (maybe except some cornercases)
handleLocal set the Endpoint to externalIP if it was not set.
externalIP comes from the public IP returned by getIP, so as soon as the node has an IP, it will have and Endpoint and be considered valid by the other nodes.

What am I missing ?

So I proposed a version of feature2 in #146

[squat]

@JulienVdG yes you're right: because of https://github.com/squat/kilo/blob/main/pkg/mesh/discoverips.go#L158, if the node doesn't have a public IP, it will advertise its private IP address as its endpoint. So you're right, we actually already have feature 1 working!

[squat]

This is now complete, thanks to @JulienVdG in #146!