net: add option to make vmnet the default route

[sakkyoi]

Description

Fix network routing issues when using multiple network interfaces with bridged mode.

Problem

When starting Colima with bridged networking mode, the routing table may not be configured correctly, leading to connectivity issues between containers and the host network.

Example before fix:

$ colima start --network-address --network-mode bridged
$ colima ssh
$ ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.1 metric 200 
default via 10.10.20.10 dev col0 proto dhcp src 10.10.20.22 metric 300 
10.10.20.0/24 dev col0 proto kernel scope link src 10.10.20.22 metric 300 
...

if interested in what happens, tcpdumps is here

ping from same subnet (works):

$ tcpdump -i col0 arp or icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on col0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:38:50.223964 IP 10.10.20.20 > 10.10.20.22: ICMP echo request, id 22125, seq 0, length 64
15:38:50.224077 IP 10.10.20.22 > 10.10.20.20: ICMP echo reply, id 22125, seq 0, length 64
...

ping from different subnet (does not work):

$ tcpdump -i col0 arp or icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on col0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:40:24.155883 IP 10.10.11.27 > 10.10.20.22: ICMP echo request, id 51225, seq 0, length 64
15:40:25.160998 IP 10.10.11.27 > 10.10.20.22: ICMP echo request, id 51225, seq 1, length 64
15:40:26.168127 IP 10.10.11.27 > 10.10.20.22: ICMP echo request, id 51225, seq 2, length 64
15:40:27.171187 IP 10.10.11.27 > 10.10.20.22: ICMP echo request, id 51225, seq 3, length 64

There are no replies because the default route for col0 has a higher metric (300) than eth0 (200), causing traffic to prefer eth0 over col0.

$ tcpdump -i eth0 arp or icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
15:40:53.436768 IP 10.10.20.22 > 10.10.11.27: ICMP echo reply, id 56345, seq 0, length 64
15:40:54.440847 IP 10.10.20.22 > 10.10.11.27: ICMP echo reply, id 56345, seq 1, length 64
15:40:55.445638 IP 10.10.20.22 > 10.10.11.27: ICMP echo reply, id 56345, seq 2, length 64
15:40:56.450777 IP 10.10.20.22 > 10.10.11.27: ICMP echo reply, id 56345, seq 3, length 64

And eth0 is not able to reach subnet outside 192.168.5.0/24.

The problem here is that the default route for col0 has a higher metric (300) than eth0 (200), causing traffic to prefer eth0 over col0.
So, requests coming into col0 would not be replied from col0 by default if the remote host is not in the same subnet.

Observed behavior

Working cases (metric adjusted):

default via 10.10.20.10 dev col0 proto dhcp src 10.10.20.22 metric 100 
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.1 metric 200 
...
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.1 metric 200 

• Requests prefer col0 when the destination is outside 192.168.5.0/24.

NAT mode (shared or without --network-address)

default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.1 metric 200 
default via 192.168.106.1 dev col0 proto dhcp src 192.168.106.2 metric 300 

or

default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.1 metric 200 

• Everything works fine because the remote host is replaced with the NAT gateway.

Solutions

• The easiest way is to adjust the metric of col0 to be lower than eth0 (e.g. 100).

Note

This change does not affect the existing behavior for networks using only the default NAT interface (eth0).

• Traffic within the 192.168.5.0/24 subnet will still go through eth0 as before.
• Only the default route metric for col0 will be affected.

This change will ensure that traffic to and from the col0 interface is properly routed, allowing for seamless communication between containers and the host network.

Open to discussion if there’s a more robust approach.

[abiosoft]

The behaviour was similar to this before i.e. the col0 interface being preferred to eth0.

However, some issues were reported (usually VPN related) when network address is enabled. Making eth0 the default route provided a consistent experience (with or without network address) and seemed to resolve the issues.

Due to that, I am leaning towards leaving the current behaviour and make it opt-in for col0 to be made the default route.

[sakkyoi]

The previous solution I used was to add a CONNMARK to packets coming from col0, allowing return traffic to follow the original path. This is admittedly a bit more complex, but it might be another approach worth considering. More information on the implementation can be found here (not work for http, just icmp):
https://gist.github.com/sakkyoi/e2a1a33709fe8aac7009bcf23fe1570a

Another idea is to replace the NAT(eth0) with the vmnet bridge (col0) if network address is enabled. I'm not fully confident about how this might affect other components, so it could be worth discussing as a potential direction.

[sakkyoi]

Also, I've adjusted the implementation of this PR to make the behavior opt-in by introducing a --network-preferred-route flag for now, though I'm not entirely sure if the flag name is the best fit.

[abiosoft]

There is one more thing to do, kindly modify the embedded config file to include the corresponding section for the config.

[sakkyoi]

Sorry, I missed that.

[abiosoft]

Thanks :)

[a0s]

Hi! I am not sure this is the best place to ask my question, but seems relevant.
Is it possible to remove or just do not create eth0 on colima start? I want to have bridge (col0) only.

[abiosoft]

Hi! I am not sure this is the best place to ask my question, but seems relevant. Is it possible to remove or just do not create eth0 on colima start? I want to have bridge (col0) only.

@a0s it is not supported at the moment.
The provisioning of the virtual machine requires an active network interface. The external address is not guaranteed to be successful, and bridge mode is even less guaranteed.