eBPF code to intercept WireGuard (/tunnels) connections

[gustavo-iniguez-goya]

Tested only on 5.10.x, 5.8.x, 5.4.x and 4.19.x, and only for x86_64 and IPv4 for now.

procmon/parse.go:FindProcess() must be edited in order to pass some checks, and in these cases maybe we could set the name (path of the process internally) as "Kernel connection", and arguments the value of /proc//comm.

Anyway, I think that the comm value should be set from the eBPF data, instead of reading it from disk again.

SEC("kprobe/iptunnel_xmit")
int kprobe__iptunnel_xmit(struct pt_regs *ctx)
{
	#ifdef OPENSNITCH_x86_32
		struct sk_buff *skb = (struct sk_buff *)(ctx->regs[2]);
	        u32 src = (u32)(ctx->regs[3]);
	        u32 dst = (u32)(ctx->regs[4]);
	#else
		struct sk_buff *skb = (struct sk_buff *)PT_REGS_PARM3(ctx);
	        u32 src = (u32)PT_REGS_PARM4(ctx);
        	u32 dst = (u32)PT_REGS_PARM5(ctx);
	#endif

	u16 sport; 
	unsigned char *head;
	u16 pkt_hdr;
	__builtin_memset(&head, 0, sizeof(head));
	__builtin_memset(&pkt_hdr, 0, sizeof(pkt_hdr));
	bpf_probe_read(&head, sizeof(head), &skb->head);
	bpf_probe_read(&pkt_hdr, sizeof(pkt_hdr), &skb->transport_header);

	struct udphdr *udph;
	struct iphdr *iph;
	__builtin_memset(&udph, 0, sizeof(udph));
	__builtin_memset(&iph, 0, sizeof(iph));

       	udph = (struct udphdr *)(head + pkt_hdr);
       	iph = (struct iphdr *)(head + net_hdr);
	bpf_probe_read(&sport, sizeof(u16), &udph->source);
	sport = (sport >> 8) | ((sport << 8) & 0xff00);

	struct udp_key_t udp_key;
	struct udp_value_t udp_value;
	u32 zero_key = 0;
	__builtin_memset(&udp_key, 0, sizeof(udp_key));
        __builtin_memset(&udp_key, 0, sizeof(udp_value));

	bpf_probe_read(&udp_key.sport, sizeof(udp_key.sport), &sport);
	bpf_probe_read(&udp_key.dport, sizeof(udp_key.dport), &udph->dest);
	bpf_probe_read(&udp_key.saddr, sizeof(udp_key.saddr), &src);
	bpf_probe_read(&udp_key.daddr, sizeof(udp_key.daddr), &dst);

	u64 *counterVal = bpf_map_lookup_elem(&udpcounter, &zero_key);
	if (counterVal == NULL){
            return 0;
        }

	struct udp_value_t *lookedupValue = bpf_map_lookup_elem(&udpMap, &udp_key);
	u64 pid = bpf_get_current_pid_tgid() >> 32;
	if ( lookedupValue == NULL || lookedupValue->pid != pid) {
		bpf_get_current_comm(&udp_value.comm, sizeof(udp_value.comm));
		udp_value.pid = pid;
		udp_value.uid = bpf_get_current_uid_gid() & 0xffffffff;
		udp_value.counter = *counterVal;
		bpf_map_update_elem(&udpMap, &udp_key, &udp_value, BPF_ANY);
		u64 newval = *counterVal + 1;
		bpf_map_update_elem(&udpcounter, &zero_key, &newval, BPF_ANY);	
	}

	/*char xx[] = "iptunnel_xmit addrs: %x -> %x\n";
	bpf_trace_printk(xx, sizeof(xx), src, dst);
	char xx1[] = "iptunnel_xmit ports: %d -> %d\n";
	bpf_trace_printk(xx1, sizeof(xx1), udp_key.sport, udp_key.dport);
	*/

	return 0;
};

[themighty1]

You didnt zero out net_hdr. Im not saying that is the reason but ebpf code is very finicky.

[gustavo-iniguez-goya]

Thank you @themighty1 for reviewing it. I've realized that net_hdr was not used. I've removed it, I've also tried to zero out udp_value and others but it doesn't make any difference. nah, solved :)

Also worth noting is that this function was included in kernel 3.8 with a different definition, later rewritten in 3.10 and 3.11 and 3.12

if included maybe we should load kprobes one by one, otherwise if this kprobe fails loading we wouldn't load the rest of kprobes.

[gustavo-iniguez-goya]

hey @themighty1 , I'd like to know your opinion on the status of this issue #513 . Do you think that it could be merged as it is right now?

On x86_64 it works on pretty much every kernel I've tried it out in. On arm64 works (with some limitations). And on i386/armhf I haven't managed to get sk_buff nor sock from registers, so the hook for these arquitectures is just ignored, or a .o module without this hook should be shipped.

[themighty1]

hi @gustavo-iniguez-goya ,nice work on that front! Yes, I think it could be merged and people can start testing it and giving feedback.

Just my guess that on other archs/distors there may be used other kernel #ifdefs which cause some fields of sk_buff to be moved to different offsets. Thus our .o is not finding any useful info at those offsets, instead finds value 0 at those offset.

[gustavo-iniguez-goya]

merged! I'm afraid that the title of the pop-ups will cause some confusion, because they are not very descriptives and don't identify clearly the process that initiated the connection, but we'll see.

thank you!

[mk-fg]

I wanted to get similar wireguard connection tracking with IPv6 support, and looking at send4/send6 in more recent 6.12 drivers/net/wireguard/socket.c, best place to get saddr/daddr/sport/dport from seem to be udp_tunnel_xmit_skb and udp_tunnel6_xmit_skb functions that they're passed to.

One quirk there is that sport/dport are passed as arguments 9/10, which need to be fetched from stack (as per System V AMD64 ABI calling convention), and with IPv6 version saddr/daddr are supposed to be set on return via pointers (e.g. saddr set to in6addr_any), so need similar cache for kretprobe as with socket connect calls.

Implemented those probes towards the end here: https://github.com/mk-fg/linux-ebpf-connection-overseer/blob/master/ebpf.c
Not sure if they might be useful in this project as well, but as I stumbled upon this discussion when wondering about why iptunnel_xmit was picked, thought to mention this alternative solution.
Seem to work for me with IPv4/IPv6 tunnels on 6.12 LTS and x86_64, but no idea about older kernels (didn't look).

[gustavo-iniguez-goya]

it would be super useful @mk-fg ! in fact, we have an issue with wireguard and ipv6, because we're not intercepting ipv6: #1250

I can test it in older kernels, 4.x, 5.x, but I guess that it should work.

[gustavo-iniguez-goya]

by the way @mk-fg , udp_tunnel* functions are only available if udp_tunnel or/and ip6_udp_tunnel modules are loaded. On Ubuntu 22 for example they're not loaded by default, only when you open a Wireguard tunnel. So in our case we'd fail adding that hook if the VPN was opened after our daemon starts.

[mk-fg]

Ah, thanks, should probably add couple modprobe lines to systemd unit as well.
Iirc IPv6 is technically also tristate in kernel and can be a module, but probably always loaded very early on due to basically anything else network-related :)