§ why?

Recently (that means on 2018-08-02), I read some interesting news on HackerNews regarding WireGuard. In particular: WireGuard getting ready for inclusion in the Linux kernel.

So I go on my merry way to replace the horrors that are OpenVPN and IPSec (Linus Torvalds, 2018-08-02).

§ config

Looks simple enough. The ArchWiki already has a page, though it is filled with abhorrent lines of iptables. Actually, the minimum viable configuration for WireGuard consists of 2 files counting 7 lines each (how it’s done is left as an exercise to the reader).

The main issue I had was understanding AllowedIPs: AllowedIPs under [Peer] has this double meaning of “I expect anything coming from that peer to have an IP address in that subnet AND I know that this subnet can only be reached through that specific peer”. Thus, AllowedIPs must not overlap.

After I figured that out, I have config files that look like this:

# Server file
[Interface]
# Which networks does my interface belong to? Notice: /24 and /64
Address = 10.5.0.1/24, 2001:470:xxxx:xxxx::1/64
PrivateKey = xxx
ListenPort = 51820

# Peer 1
[Peer]
PublicKey = xxx
# Which source IPs can I expect from that peer? Notice: /32 and /128
AllowedIps = 10.5.0.35/32, 2001:470:xxxx:xxxx::746f:786f/128

# Peer 2
[Peer]
PublicKey = xxx
# Which source IPs can I expect from that peer? This one has a LAN which can
# access hosts/jails without NAT.
# Peer 2 has a single IP address inside the VPN: it's 10.5.0.25/32
AllowedIps = 10.5.0.25/32,10.21.10.0/24,10.21.20.0/24,10.21.30.0/24,10.31.0.0/24,2001:470:xxxx:xxxx::ca:571e/128

# Peer 1 file
[Interface]
# Which networks does my interface belong to? Notice: /24 and /64
Address = 10.5.0.35/24, 2001:470:xxxx:xxxx::746f:786f/64
PrivateKey = xxx

# Server
[Peer]
PublicKey = xxx
# I want to route everything through the server, both IPv4 and IPv6. All IPs are
# thus available through the Server, and I can expect packets from any IP to
# come from that peer.
AllowedIPs = 0.0.0.0/0, ::0/0
# Where is the server on the internet? This is a public address. The port
# (:51820) is the same as ListenPort in the [Interface] of the Server file above
Endpoint = 1.2.3.4:51820
# Usually, clients are behind NAT. to keep the connection running, keep alive.
PersistentKeepalive = 15

# Peer 2 file
[Interface]
Address = 10.5.0.25/24, 2001:470:xxxx:xxxx::ca:571e/64
PrivateKey = xxx
# Peer 2 is in a DMZ, so I can set the ListenPort - though it's not required
ListenPort = 51820

# Server
[Peer]
PublicKey = xxx
# I know which networks are available on the server: the VPN network
# (10.5.0.0/24) and lots of internal stuff. But I don't want to tunnel
# everything, so I don't use 0.0.0.0/0
AllowedIps = 10.5.0.0/24,10.10.10.0/24,10.10.20.0/24,10.10.30.0/24,10.10.40.0/24,2001:470:xxx::0/48
# Where is the server?
Endpoint = 1.2.3.4:51820

I became fluent so fast that I could write the wiki page for archlinux.fr.

§ bugs

Of course, it would have been fine for anyone other than me. I had to find two bugs in this battle-tested piece of software. (It reminds me that I didn’t write about that OpenVPN bug I found a while back: I really do have a super power)

§ FreeBSD went haywire with IPv6 routes

# #do some configuration changes...
# wg-quick down wg0 && wg-quick up wg0

Everything looks good…? not?… IPv6 connectivity is down.

client# ping 2001:470:xxxx:xxxx::1
server# tcpdump -tttnei wg0 host 2001:470:xxxx:xxxx::1
 00:00:00.000000 AF IPv6 (28), length 108: 2001:470:xxxx:xxxx::746f:786f > 2001:470:xxxx:xxxx::1: ICMP6, echo request, seq 1, length 64
 00:00:01.002025 AF IPv6 (28), length 108: 2001:470:xxxx:xxxx::746f:786f > 2001:470:xxxx:xxxx::1: ICMP6, echo request, seq 2, length 64

OK, there are only requests, no replies, even though I’m sure it wasn’t broken. Bring everything down wg-quick down wg0, ask on IRC, fiddle with the configuration, bring the VPN back up wg-quick up wg0. That works. I’m now a buffon on #wireguard@Freenode. Add some more peers, wg-quick down wg0 && wg-quick up wg0, and now it’s broken again.

What could be wrong? IP addresses? Nah, they look good.

# ifconfig
wg0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 10.5.0.1 --> 10.5.0.1 netmask 0xffffff00
    inet6 fe80::eea8:6bff:fef1:c8a3%wg0 prefixlen 64 scopeid 0x9
    inet6 2001:470:xxxx:xxxx::1 prefixlen 64
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    groups: tun
    Opened by PID 86589

Maybe routes? It can’t be routes. The prefixlen is correctly set on the interface: I shouldn’t…

# netstat -rnfinet6 | grep 2001:470:xxxx:xxxx
2001:470:xxxx:xxxx::1             link#9                        UHS         lo0

No route to the /64: that’s the issue. What happens when I add them by hand?

server# route add -inet6 2001:470:xxxx:xxxx::/64 2001:470:xxxx:xxxx::1
server# netstat -rnfinet6|grep 2001:470:xxxx:xxxx
2001:470:xxxx:xxxx::/64           link#9                        U           wg0
2001:470:xxxx:xxxx::1             link#9                        UHS         lo0
client# ping 2001:470:xxxx:xxxx::1
server# tcpdump -tttnei wg0 host 2001:470:xxxx:xxxx::1
 00:00:00.000000 AF IPv6 (28), length 108: 2001:470:xxxx:xxxx::746f:786f > 2001:470:xxxx:xxxx::1: ICMP6, echo request, seq 1, length 64
 00:00:00.000031 AF IPv6 (28), length 108: 2001:470:xxxx:xxxx::1 > 2001:470:xxxx:xxxx::746f:786f: ICMP6, echo reply, seq 1, length 64

That’s more like it! Now we have ping replies. OK, so routes aren’t added correctly. Sometimes.

Turns out it’s a race condition in wg-quick for FreeBSD.

Solution: just have to sleep(1) a bit before you can wg-quick up. Thus the current work in progress for an rc.d script.

§ Android

Turns out Android doesn’t like you messing too much with routes either. OpenVPN on, off, WireGuard on, off, on, OpenVPN on… oh and now you don’t get IPv6 in your tunnel. And don’t bother checking routes on the console either, ip -6 ro is broken and will stay completely mute.

Solution: a bit of magic.