Tagged in: projects, networking

I finally got real networking equipment^TM at my new place, so I took the time to move away from the flimsy ISP box I am provided with to do some interesting networking. Of course, if I need to write an article about it: it means it wasn’t all smooth sailing.

§ very bad hardware

For the very specific purpose of replacing my Livebox, I had purchased a chinese machine with 2 NICs a few years ago. I installed OpenBSD on that aluminium box.

Well, I’m not doing it again.

§ that wouldn’t boot

The machine would sometimes not boot and stay stuck at the OpenBSD boot prompt.

boot>

But there’s a remedy to that, it’s all in boot(8):

# echo "boot" > /etc/boot.conf

Fixed!

§ with very bad throughput

obsd% iperf -c 192.168.1.110 -P4
------------------------------------------------------------
Client connecting to 192.168.1.110, TCP port 5001
TCP window size: 55.1 KByte (default)
------------------------------------------------------------
[  5] local 192.168.1.24 port 32286 connected with 192.168.1.110 port 5001
[  6] local 192.168.1.24 port 34917 connected with 192.168.1.110 port 5001
[  3] local 192.168.1.24 port 22717 connected with 192.168.1.110 port 5001
[  4] local 192.168.1.24 port 43039 connected with 192.168.1.110 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   217 MBytes   182 Mbits/sec
[  6]  0.0-10.0 sec   188 MBytes   158 Mbits/sec
[  3]  0.0-10.0 sec   134 MBytes   112 Mbits/sec
[  4]  0.0-10.0 sec  94.1 MBytes  78.9 Mbits/sec
[SUM]  0.0-10.0 sec   633 MBytes   531 Mbits/sec

I have another machine on the same network (same switch, same cables, etc.):

fbsd% iperf -c 192.168.1.110 -P4
------------------------------------------------------------
Client connecting to 192.168.1.110, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  2] local 192.168.1.100 port 58597 connected with 192.168.1.110 port 5001
[  1] local 192.168.1.100 port 51302 connected with 192.168.1.110 port 5001
[  4] local 192.168.1.100 port 25458 connected with 192.168.1.110 port 5001
[  3] local 192.168.1.100 port 20018 connected with 192.168.1.110 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3] 0.00-10.07 sec   275 MBytes   229 Mbits/sec
[  4] 0.00-10.07 sec   278 MBytes   232 Mbits/sec
[  1] 0.00-10.07 sec   287 MBytes   239 Mbits/sec
[  2] 0.00-10.07 sec   286 MBytes   238 Mbits/sec
[SUM] 0.00-10.02 sec  1.10 GBytes   943 Mbits/sec

Practically half speed sad face.

The very bad throughput is due to the machine running OpenBSD and the NICs being Realtek:

# dmesg
[...]
ppb0 at pci0 dev 28 function 0 "Intel Braswell PCIE" rev 0x21: msi
pci1 at ppb0 bus 1
re0 at pci1 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL (0x2c80), msi, address ...
rgephy0 at re0 phy 7: RTL8169S/8110S/8211 PHY, rev. 5
ppb1 at pci0 dev 28 function 1 "Intel Braswell PCIE" rev 0x21: msi
pci2 at ppb1 bus 2
re1 at pci2 dev 0 function 0 "Realtek 8168" rev 0x06: RTL8168E/8111E-VL (0x2c80), msi, address ...

I’m not compromising on running OpenBSD (because pf(4) is just awesome), so I’ll be hunting for better hardware (Intel NICs specifically).

Unfixed for now.

§ and a quantum interface

You know how observing a (quantum) event changes the outcome? Well guess what: it also happens with network flows. Just as I was closing all the things I had opened to connect (!) to the Orange network, it all broke. No ping(8), no nothing. Relaunch dhclient(8), no reply. Fire up tcpdump(8), find out what’s happening. Now it works, weird. Close tcpdump(8), it all stops again.

Uh oh.

So yeah: my network interface (specifically egress) doesn’t actually handle hardware VLAN tagging correctly.

% ifconfig re0 hwfeatures
re0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 6000
        hwfeatures=8037<CSUM_IPv4,CSUM_TCPv4,CSUM_UDPv4,VLAN_MTU,VLAN_HWTAGGING,WOL> hardmtu 6122
        lladdr ...
        description: port cote micro UPSTREAM
        index 1 priority 0 llprio 3
        media: Ethernet autoselect (1000baseT full-duplex)
        status: active

tcpdump(8) needs to run for the interface to work; we launch it automatically at startup:

# grep tcpdump /etc/rc.conf.local
tcpdump -i vlan832 host 123.123.123.123 >/dev/null 2>&1 &

§ super duper weird ISP

In its infinite wisdom, Orange France does not comply with numerous RFCs, so we need to get creative. First, upstream is on VLAN 832; second, authentication requires we pass hand-crafted hex streams; third, priority and QoS.

§ using VLAN832

That was easy, and documented everywhere.

parent re0 vnetid 832
description "ISP link"
up
!dhclient vlan832 &

Also, we override the hardware (MAC) address of the NIC with that of the Livebox we’re replacing.

# This is the physical upstream link
# lladdr is from the Livebox4
lladdr "78:81:..."
up

§ requiring IPv4 authentication

NB: dhclient(8) was crippled by upgrading to OpenBSD 7.2. The advice remains, but a working setup for OpenBSD ≥ 7.2 can be found in a later paragraph.

Orange requires we use the authentication options in DHCP requests. This is option-90 in DHCPv4. Orange disregards replay detection mechanisms and requires we craft a special payload that depends on our FTI user and password (that password being available in print on the contract I signed with Orange; it seems impossible to extract it from anywhere else: Livebox, “customer space” at orange.fr, etc.). The easiest way to update a config file regularly is to use cat(1) in the system’s crontab(5):

obsd# crontab -l
FTI_USER=fti/...
FTI_PASS=...
# DHCP woes
~       0       *       *       0       /usr/local/bin/orange_hexauth > /etc/orange_hexauth 2>/dev/null
~       1       *       *       0       cat /etc/dhclient.conf.head /etc/orange_hexauth /etc/dhclient.conf.tail > /etc/dhclient.conf
~       2~5     *       *       0       dhclient vlan832

The orange_hexauth script is available in my git repo. I can’t thank the people on lafibre.info enough for their work on reversing the generation of said option.

An (unexpected) additional challenge was to remove the last \n (newline) at the end of /etc/dhclient.conf.head. Because the hex string has to be in the “middle” of a line, the last char at the end of the first file we concatenate must not be a newline. We use dd(1) for that.

dest=/etc/dhclient.conf.head
dd if=/dev/null of="$dest" obs="$(( $(wc -c < "$dest") -1 ))" seek=1

interface "vlan832" {
  send dhcp-class-identifier "sagem";
  send user-class "+FSVDSL_livebox.Internet.softathome.Livebox4";
  ignore host-name ;
  #No newline at the end of the file!
  send option-90 00:00:00:00:00:00:00:00:00:00:00:

;
  request;
  request subnet-mask, routers, domain-name-servers, domain-name, broadcast-address, dhcp-lease-time, dhcp-renewal-time, dhcp-rebinding-time, option-90, domain-search, option-120, option-125;
}

Fun fact: OpenBSD’s crontab(5) syntax allows for the special char ~ to be used as a shorthand for “any valid random value”. vim(1) wasn’t coloring the file correctly, but that was nothing a quick patch couldn’t fix.

§ and IPv6 authentication

Now for the fun part. IPv6 is the future^TM and to do it properly, I had wanted to do proper prefix delegation to a router that would hand out real public IPv6 addresses to all devices home, depending on where they are in the network (safe network, guest network, etc.).

The Livebox doesn’t do that. (And this is actually why I even wanted to do what I’m writing about right now.) It can’t manage more than two networks (home and guests) and it doesn’t do prefix delegation properly.

After fighting quite a bit to get a DHCPv6 client to send out “spoofed” sollicit requests by capturing my OpenBSD’s network chatter and comparing it to that I captured from the Livebox, I finally managed it with isc-dhcp-client. Hooray, end of story.

ISC has ended development on the ISC DHCP client as of early 2022. This client implementation is no longer maintained and should not be used in production any longer.

ISC website archived on 2022-09-04

Uh oh.

I can successfully spoof the DHCPv6 requests so that fe80::ba0:bab hands me out a /56. This means that I’m not at a dead end. (Success?)
I need to move to another DHCPv6 client.

Dibbler was recommended regularly on lafibre.info, but it too has been discontinued (in 2017). OpenBSD doesn’t ship a DHCPv6 client capable of authentication. I’m left only with dhcpcd.

After trying to use my hex auth string in dhcpcd-v4.9.1, I found out that it’s only supported after commit 4b37f00. The rest is then pretty uneventful. git clone, configure, make, make install; get the rc.d(8) service file from the port; dhcpcd works as expected; rad(8) worked like a charm (can you even believe it doesn’t need explicit definitions of the current subnet in the config at all?).

# only handles IPv6
noipv6rs
ipv6only
nohook resolv.conf hostname ntp.conf
allowinterfaces vlan832
debug

# based on https://blog.brimbelle.org/index.php/2018/04/30/fibre-orange-ipv6-et-dhcpcd/
interface vlan832
        # 0003001<MAC_ADDRESS> in /var/db/dhcpcd/duid
        # no other option necessary here
        # iaid below = last 4 bytes of the lladdr of the livebox
        iaid    01234567
        # delegate /64s to all interfaces. rad(8) will handle them
        ia_pd   01234567 vlan49//64 vlan50//64 vlan51//64 re1//64
        option auth
        # This userclass below only works with dhcpcd (it prepends 00:2b)
        userclass FSVDSL_livebox.Internet.softathome.Livebox4
        vendclass 1038 sagem
        authprotocol token 0x123/0x456
        # This authtoken 0x456 below is a magic string (dhcplivebox250) that Orange returns
        authtoken 0x456 "" forever 64:68:63:70:6c:69:76:65:62:6f:78:66:72:32:35:30
        # Very important: no newline at the end of the file! Only a space!
        # Very important: only works with dhcpcd > 4.9.1
        authtoken 0x123 "" forever

# Look! no subnet definition!
interface vlan49 {
        dns {
                nameserver 2606:4700:4700::1112
        }
}

interface vlan50 {
        dns {
                nameserver 2606:4700:4700::1112
        }
}

# IoT -- need an IP, but no DNS (at the moment)
interface vlan51

interface re1 {
        dns {
                nameserver 2606:4700:4700::1112
        }
}

obsd# crontab -l
FTI_USER=fti/...
FTI_PASS=...
# DHCP woes
~       0       *       *       0       /usr/local/bin/orange_hexauth > /etc/orange_hexauth 2>/dev/null
~       1       *       *       0       cat /etc/dhclient.conf.head /etc/orange_hexauth /etc/dhclient.conf.tail > /etc/dhclient.conf
~       1       *       *       0       cat /etc/dhcpcd.head /etc/orange_hexauth > /etc/dhcpcd.conf
~       2~5     *       *       0       dhclient vlan832
~       2~5     *       *       0       dhcpcd -n

We can check that dhcpcd(8) works as expected:

Sep 26 05:53:01 rutledge dhcpcd[76422]: sending signal HUP to pid 97
Sep 26 05:53:01 rutledge dhcpcd[87647]: received SIGHUP, rebinding
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan832: config file changed, expiring leases
Sep 26 05:53:01 rutledge dhcpcd[87647]: re1: deleting address 2a01:xxxx:xxxx:xxx2::1/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: re1: deleting route to 2a01:xxxx:xxxx:xxx2::/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan49: deleting address 2a01:xxxx:xxxx:xxx7::1/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan49: deleting route to 2a01:xxxx:xxxx:xxx7::/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan50: deleting address 2a01:xxxx:xxxx:xxx8::1/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan50: deleting route to 2a01:xxxx:xxxx:xxx8::/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan51: deleting address 2a01:xxxx:xxxx:xxx9::1/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan51: deleting route to 2a01:xxxx:xxxx:xxx9::/64
Sep 26 05:53:01 rutledge dhcpcd[87647]: lo0: deleting reject route to 2a01:xxxx:xxxx:xxx0::/56 via ::1
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan832: IAID 01:23:45:67
Sep 26 05:53:01 rutledge dhcpcd[87647]: vlan832: rebinding prior DHCPv6 lease
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan832: REPLY6 received from fe80::ba0:bab
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan832: renew in 85536, rebind in 207360, expire in 259200 seconds
Sep 26 05:53:02 rutledge dhcpcd[87647]: lo0: adding reject route to 2a01:xxxx:xxxx:xxx0::/56 via ::1
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan832: delegated prefix 2a01:xxxx:xxxx:xxx0::/56
Sep 26 05:53:02 rutledge dhcpcd[87647]: re1: adding address 2a01:xxxx:xxxx:xxx2::1/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan49: adding address 2a01:xxxx:xxxx:xxx7::1/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan50: adding address 2a01:xxxx:xxxx:xxx8::1/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan51: adding address 2a01:xxxx:xxxx:xxx9::1/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: re1: adding route to 2a01:xxxx:xxxx:xxx2::/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan49: adding route to 2a01:xxxx:xxxx:xxx7::/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan50: adding route to 2a01:xxxx:xxxx:xxx8::/64
Sep 26 05:53:02 rutledge dhcpcd[87647]: vlan51: adding route to 2a01:xxxx:xxxx:xxx9::/64

The last issue was no default IPv6 route:

route add ::/0 -inet6 fe80::ba0:bab%vlan832

FWIW, the /56 that Orange delivers can be split into 256 /64 (=2^64-56). Not really ideal in the IPv6 world, but clearly enough for our usecase. I can even deliver a few /64s to my FreeBSD NAS at home for each of its own subnets.

A quick and dirty hook for dhcpcd(8) follows (see dhcpcd-run-hooks(8)):

#!/bin/sh

# For use on Orange France network

if [ "$interface" = "vlan832" ]; then
  case "$reason" in
    # Cannot use BOUND6 because vlan832 doesn't get its own IPv6 address
    BOUND)   route add ::/0 -inet6 fe80::ba0:bab%$interface ;;
    STOPPED) route del ::/0 -inet6 fe80::ba0:bab%$interface ;;
  esac
fi

§ and OpenBSD decided dhclient(8) was no longer necessary

From the 7.2 release notes:

Changed dhclient(8) to defer to dhcpleased(8) by doing execve ifconfig and providing syslog warnings about deprecated options.

Meaning that I ended up with a crippled DHCPv4 client (dhcpleased.conf(5) doesn’t handle auth at all).

Uh oh.

As a result, I fell back to dhcpcd(8) and let it also handle IPv4:

# handles both IPv4 and IPv6
noipv6rs
noipv4ll
nohook hostname ntp.conf
allowinterfaces vlan832
debug

# based on https://blog.brimbelle.org/index.php/2018/04/30/fibre-orange-ipv6-et-dhcpcd/
interface vlan832
        # 0003001<MAC_ADDRESS> in /var/db/dhcpcd/duid
        # no other option necessary here
        # iaid below = last 4 bytes of the lladdr of the livebox
        iaid    01234567
        # delegate /64s to all interfaces. rad(8) will handle them
        ia_pd   01234567 vlan49//64 vlan50//64 vlan51//64 re1//64
        option auth
        # This userclass below only works with dhcpcd (it prepends 00:2b)
        userclass FSVDSL_livebox.Internet.softathome.Livebox4
        vendclass 1038 sagem
        authprotocol token 0x123/0x456
        # This authtoken 0x456 below is a magic string (dhcplivebox250) that Orange returns
        authtoken 0x456 "" forever 64:68:63:70:6c:69:76:65:62:6f:78:66:72:32:35:30
        # Very important: no newline at the end of the file! Only a space!
        # Very important: only works with dhcpcd > 4.9.1
        authtoken 0x123 "" forever

rcctl enable dhcpcd

# crontab -l
# DHCP
~       0       *       *       0       umask 077; /usr/local/bin/orange_hexauth > /etc/orange_hexauth 2>/dev/null
~       1       *       *       0       cat /etc/dhcpcd.conf.head /etc/orange_hexauth > /etc/dhcpcd.conf
~       2       *       *       0       dhcpcd -n
# Bad hardware, really, readers shouldn't need that
@reboot /usr/sbin/tcpdump -ni vlan832 -w /dev/null host 123.123.123.123 2>/dev/null &

parent re0 vnetid 832
description "ISP link"
up

Don’t forget the dhcpcd(8) hook.

§ as well as specifics for priority

❤️ pf.conf(5)

The pf ruleset is whatever you need it to be, but Orange-specific rules are as follows:

# martians also includes IPv6 martians
table <martians> { ... }
...
# all the rest is priority 0 and TOS 0x00
match out log on egress set prio 0 tos 0x00
# DHCP packets are priority 6
match out log on egress proto { tcp udp } from (self) to 255.255.255.255 port { bootps bootpc } set prio 6
match out log on egress inet6 from (self) to ff02::1:2 set prio 6
...
pass  in  quick log inet6 from fe80::ba0:bab to (self)
...
# Martians should never be a source on packets going out
block out quick log on egress from <martians>
pass  out quick log

§ very good hardware

§ ubiquiti

For my new place, I purchased:

One USW-24-PoE switch
Three UniFi6 Lite WAPs
One UniFi6 Long-Range WAP

The three small WAPs were one for each level (basement, ground floor, upstairs) (also, partner approval factor: ✔️), the LR WAP for outdoors (my yard is all length, little breadth). The switch was installed downstairs, using all the RJ45 cabling in the walls.

Not really much to say, except that even though the management app running on my Linux desktop looks nice, it’s hungry for RAM. And the PKGBUILD could use improvements to avoid runtime errors.

● unifi.service - Ubiquiti UniFi Server
     Loaded: loaded (/usr/lib/systemd/system/unifi.service; enabled; preset: disabled)
     Active: active (running) since Fri 2022-09-16 08:09:25 CEST; 1 week 0 days ago
   Main PID: 2182815 (java)
      Tasks: 122 (limit: 38403)
     Memory: 1.2G
        CPU: 38min 8.031s
     CGroup: /system.slice/unifi.service
             ├─2182815 /usr/bin/java -jar /usr/lib/unifi/lib/ace.jar start
             └─2182876 /usr/bin/mongod --dbpath /usr/lib/unifi/data/db --port 27117 --unixSocketPrefix /usr/lib/unifi/run --logRotate reopen --logappend --logpath /usr/lib/unifi/logs/mongod.log --pidfilepath /usr/lib/unifi/run/mongod.pid --wiredTigerEngineConfigString=cache_size=256M --bind_ip 127.0.0.1

I can create 4 WiFi networks (adults, kids, guests, IoT), each mapped to a VLAN, and each of them with specific firewall rules in pf.conf(5).

§ dell optiplex 7020

After I published my article, I made some changes to the machine: I scored a 85€ (70€ + shipping) refurbished Dell OptiPlex 7020, and slapped an Intel 82576 NIC in it. #openbsd@libera.chat suggested that the em(4) driver would perform much better.

hw.model=Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz
hw.product=OptiPlex 7020
hw.vendor=Dell Inc.

em0 at pci0 dev 25 function 0 "Intel I217-LM" rev 0x04: msi, address ...
em1 at pci2 dev 0 function 0 "Intel 82576" rev 0x01: msi, address ...
em2 at pci2 dev 0 function 1 "Intel 82576" rev 0x01: msi, address ...

obsd% iperf -c 192.168.1.111 -P4
------------------------------------------------------------
Client connecting to 192.168.1.111, TCP port 5001
TCP window size: 17.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.1.1 port 7912 connected with 192.168.1.111 port 5001
[  3] local 192.168.1.1 port 44204 connected with 192.168.1.111 port 5001
[  4] local 192.168.1.1 port 8877 connected with 192.168.1.111 port 5001
[  6] local 192.168.1.1 port 46663 connected with 192.168.1.111 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   222 MBytes   186 Mbits/sec
[  3]  0.0-10.0 sec   273 MBytes   229 Mbits/sec
[  4]  0.0-10.0 sec   321 MBytes   269 Mbits/sec
[  6]  0.0-10.0 sec   296 MBytes   248 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   932 Mbits/sec

Maybe the next step is to plug the fiber directly in my switch instead of the pointless external adaptator.

§ see also

my forum post on lafibre.info in French

Securing home (part 2)

Or Puffy returns 🐡

§ very bad hardware

§ that wouldn’t boot

§ with very bad throughput

§ and a quantum interface

§ super duper weird ISP

§ using VLAN832

§ requiring IPv4 authentication

§ and IPv6 authentication

§ and OpenBSD decided dhclient(8) was no longer necessary

§ as well as specifics for priority

§ very good hardware

§ ubiquiti

§ dell optiplex 7020

§ see also

§ contact

§ legal