IncusOS Router / Server

ℹ️ This is the first of potentially many posts that I’m trying to collate and write after I’ve completed a project. It’s only been a few weeks this time, so hopefully I can remember most of what I did, but if anything seems a little disjointed that’s probably why.

Open Table of contents

Background
- Network Topology
IncusOS Install
Network Configuration
OpenWRT Install
- Instance Creation
- OpenWRT Config
Other Services
Update: Troubleshooting

Background

I’d been running OpenWRT for ages on some Mikrotik hardware, but was occasionally frustrated when some package or other wasn’t available for the mips architecture, or didn’t run well due to a lack of processing power. I have an i3-based home server for all my service containers, but decided something a bit more capable was called for for my router, and bought an N150-based Mini PC.

Installing OpenWRT directly on the mini PC would have been the most straightforward, but then I would be beholden to opkg package availability for anything else I wanted to install. Instead I decided to install Debian, and run OpenWRT in a container. Rather than dealing with Docker though, I decided to use Incus as it allows for a lower level of integration with its systems containers.

This worked great for months, and then recently IncusOS reached a point where it seemed stable enough for me to try as a replacement for Debian in this case. IncusOS is an immutable OS (basically you can’t modify any of its system binaries) whose whole purpose is just running Incus (which is all I really needed for OpenWRT and a handful of support containers).

Network Topology

For reference, my network is divided into three VLANS:

1: LAN (full access for servers and trusted machines)
2: IoT (Internet but no LAN access for IoT devices and guests)
3: Cameras (No Internet or LAN access by default)

I have a server running docker that’s connected to all three VLANS, and a WiFi AP that broadcasts SSIDs for each VLAN.

The mini PC has 4 ports, which I will be using for:

1: wan to cable modem
2: buwan (empty for now, but eventually backup WAN)
3: A VLAN-aware trunk port to the network switch
4: A VLAN-naive port for access to the IncusOS interface

IncusOS Install

Image Download

ℹ️ When I went through this process the IncusOS documentation was still in an earlier stage, but is now much more complete. You should follow the up-to-date instructions there for your install, but I will record here anything I did that seemed noteworthy for my setup.

First on my Windows desktop in WSL2 I installed the Incus client using apt install incus, and ran incus remote get-client-certificate to get the initial certificate.

Using the IncusOS image downloader I selected the USB image type, selected Wipe the target drive, and though I selected Automatically reboot after installation I sort of recommend against it because sometimes the machine would reboot back into the installer instead of off the installed OS. Downloaded and flashed the image onto a USB drive using Rufus, and plugged it into the Mini PC

⚠️ Internet/DNS Access!

Parts of the IncusOS boot networking process require access to the Internet. They have relaxed the requirements a bit for subsequent boots, but for the first boot it will need to be able to connect to a DNS server, resolve linuxcontainers.org, and check there for updates. It also wants to connect to an NTP server, but seems okay if it can’t.

For this reason I connected Port 4 (the “management” port) to my old Mikrotik router which I connected to the cable modem. The Mikrotik router will assign an IP to the management port via DHCP and provide DNS and routing to the Internet for setup.

Once the installation is successful, IncusOS should boot without Internet access, but you may need to recreate that setup for troubleshooting.

UEFI Secure Boot

I initially had some weird issues booting the installer, and resolved the issue by going into the UEFI firmware settings and setting Secure Boot (which needs to be on) back to Setup Mode. This varies by firmware, and I don’t recall exactly how I did it (or even which of the multiple attempts and reboots was the successful one), but it was an important step for me.

Network Configuration

ℹ️ There are probably better ways to do this, but for my initial install I just connected my PC directly to the Mikrotik router as well so that I could access Incus via Port 4.

After IncusOS was installed (and the encryption recovery key copied down, of course), and I’d used incus remote add to add my IncusOS machine in WSL2, I ran incus admin os system edit IncusOS:network to configure the network. My full config (obfuscated) is below, and then I’ll touch on some of the important bits for my setup.

config:
  dns:
    domain: mydomain.cloud
    hostname: incus
    nameservers:
    - 10.14.0.53
    - 192.168.1.1
    - fd10::d0cc:53
    search_domains:
    - mydomain.cloud
  interfaces:
  - hwaddr: aa:bb:cc:dd:ee:bb
    lldp: false
    name: wan
    required_for_online: "no"
  - hwaddr: aa:bb:cc:dd:ee:bc
    lldp: false
    name: buwan
    required_for_online: "no"
  - hwaddr: aa:bb:cc:dd:ee:bd
    lldp: false
    name: lan
    required_for_online: "no"
    roles:
    - instances
    vlan_tags:
    - 1
    - 2
    - 3
  - addresses:
    - 192.168.1.14/24
    - fd10::14/64
    - slaac
    hwaddr: aa:bb:cc:dd:ee:be
    lldp: false
    name: mgmt
    required_for_online: "no"
    roles:
    - management
    routes:
    - to: 0.0.0.0/0
      via: 192.168.1.1
    - to: ::/0
      via: fd10::1

Getting the network config right was by far the most time-consuming part of the process, but I’m hoping this post will help speed along anyone trying a similar setup (and a huge thanks to Stéphane for helping me with this).

config:
  dns:
    # ...
    nameservers:
    - 10.14.0.53 # AdGuard Home
    - 192.168.1.1 # Mikrotik Router
    - fd10::d0cc:53 # Backup docker DNS

I have nameservers explicitly configured because until the OpenWRT container is running (after IncusOS boots), IncusOS won’t have an easy way to automatically discover a DNS server. Before the requirements for DNS-checking were relaxed, I used a backup instance of AdGuard Home with an explicit linuxcontainers.org rewrite to trick IncusOS into thinking it had access to DNS. While I don’t think this is required anymore, it’s probably not a bad idea to at least make sure these are configured to something you know will be accessible.

interfaces:
  - hwaddr: aa:bb:cc:dd:ee:bb
    lldp: false
    name: wan
    required_for_online: "no"
  - hwaddr: aa:bb:cc:dd:ee:bc
    lldp: false
    name: buwan
    required_for_online: "no"

The first two ports are going to be passed directly to OpenWRT, so aren’t given any roles or addresses.

- hwaddr: aa:bb:cc:dd:ee:bd
    lldp: false
    name: lan
    required_for_online: "no"
    roles:
    - instances
    vlan_tags:
    - 1
    - 2
    - 3

The third port is also going to be passed to OpenWRT, but needs to be aware of the vlan_tags. I’ve assigned the instances role as this will allow me to create Incus bridge networks connected to any of the VLANs.

- addresses:
    - 192.168.1.14/24
    - fd10::14/64
    - slaac # Helps with Internet access
    hwaddr: aa:bb:cc:dd:ee:be
    lldp: false
    name: mgmt
    required_for_online: "no"
    roles:
    - management
    routes:
    - to: 0.0.0.0/0
      via: 192.168.1.1 #OpenWRT
    - to: ::/0
      via: fd10::1 #OpenWRT

The fourth and final port has static IP addresses assigned, the management role, and static routes to OpenWRT. The static IP allows me to connect to the Incus management interface even if the OpenWRT container has failed to launch for some reason. Adding slaac can help with providing a PD assigned IPv6 and access to the Internet.

Saving this configuration will cause Incus to run its network checks again (i.e. DNS needs to work correctly still), and the commit the network configuration.

ℹ️ Once or twice I ran into an issue where the network wasn’t working correctly and running incus admin os system edit IncusOS:network and saving the file without changes to force a reconfiguration seemed to resolve whatever the issue was.

OpenWRT Install

Instance Creation

Installing OpenWRT on Incus was a breeze because LXC maintains their own image on their image server. As with IncusOS the tricky part was networking, though once IncusOS is configured it’s more straightforward for the OpenWRT image.

I created an Incus profile to contain the networks, which I recommend because if you need to switch to a different OpenWRT instance or recreate it, you don’t need to re-do this part of the config at least (though with fixed MAC addresses, you probably shouldn’t try to apply this profile to multiple instances).

name: router-network
description: Network ports for router (fixed MAC addresses)
devices:
  cams:
    hwaddr: 10:66:6A:12:34:DD
    name: cams
    nictype: bridged
    parent: lan
    type: nic
    vlan: '3'
  iot:
    hwaddr: 10:66:6A:12:34:CC
    name: iot
    nictype: bridged
    parent: lan
    type: nic
    vlan: '2'
  lan:
    hwaddr: 10:66:6A:12:34:BB
    name: lan
    nictype: bridged
    parent: lan
    type: nic
    vlan: '1'
  wan:
    hwaddr: 10:66:6A:12:34:AA
    name: wan
    nictype: bridged
    parent: wan
    type: nic
config: {}
project: default

OpenWRT is pre-configured to expect an interface named eth0, so it might not connect up to the provided devices by itself. You can either use the Incus terminal to edit /etc/config/network in the instance to configure at least lan so that you can access OpenWRT from your PC, or (and I think this is probably better) you can create a proxy in Incus:

devices:
  proxy-1:
    bind: host
    connect: tcp::80
    listen: tcp:192.168.1.14:8888
    type: proxy

This will let you connect to OpenWRT essentially from its own localhost via Incus’s IP and a unique port (192.168.1.14:8888) for me. This came in handy more than once when I messed up something about OpenWRT’s network configuration.

❔ In hindsight I think I should have configured an Incus storage volume to mount at /etc/config (or /etc?) so that configuration could be more easily persisted between instances or backed up. However the normal configuration backup process for OpenWRT still works fine.

OpenWRT Config

Configuring OpenWRT was pretty straightforward from here. Each of the configured VLANs shows up as a separate network device so there’s no VLAN config necessary in OpenWRT.

ℹ️ It’s worth reiterating that for my particular setup, Ports 1-3 are all controlled by OpenWRT, and Port 4 is assigned to Incus. There’s a sort of unusual situation where Port 3 and Port 4 are both plugged into my network switch. If Incus wants to access the Internet (or any of the VLANs) packets will actually be sent out from the mini PC (from Incus) via Port 4 to the switch, and back into the mini PC (to OpenWRT) via Port 3.

Other Services

I won’t go into all the details here, but I also installed a few other services on Incus. I created a Bridge network in Incus (with DHCP, etc. turned off):

project: default
name: incus-br0
description: ''
type: bridge
config:
  ipv4.address: none
  ipv6.address: none

and added it to my OpenWRT instance with this:

devices:
  incus-br0:
    name: incus-br0
    network: incus-br0
    security.ipv4_filtering: 'false'
    security.mac_filtering: 'false'
    type: nic

This way any additional instances I create on Incus I can just have this bridge added to them, and then OpenWRT will hand out IPs via DHCP. Because it shows up as its own network in OpenWRT, it also can easily get its own firewall zone to control access to and from instances (the tradeoff being that all traffic to/from the instances has to be routed— you can’t connect to the instances directly from e.g. LAN).

⚠️ Many OCI containers lack network configuration support and are only configured by DHCP/SLAAC. This practically means that you need either OpenWRT or Incus to have DHCP enabled on incus-br0 so that they’ll get an IP.

Currently I have the following instances (all OCI containers) also running on Incus:

Acme.sh: Provisions SSL certificates for use by Incus, OpenWRT, and other services.
AdGuard Home: Primary DNS server for my home network. There are some tricky things here because OpenWRT will be handing this out to clients on the network, but this instance might go down separate from OpenWRT. This isn't really any different than hosting a PiHole or similar though, so not a big downside.
Traefik: I'm using Traefik as a reverse proxy for accessing services on my home network from outside my home. Hosting it here in Incus means I can set up this Traefik with a simple / easy to keep secure configuration that authenticates traffic and then forwards it to the Traefik instance on my docker host where I can be more free to experiment.
Crowdsec: An extra layer of security for Traefik. On Incus for the same reasons as Traefik: I can set it up once and not worry about mistakes during reconfiguration.

Update: Troubleshooting

The IncusOS update process is designed to be automatic and safe (rolling back to a previous firmware automatically if the update fails), but because it was early in development I ran into an issue where the new version failed because of one problem, and the previous version couldn’t be rolled back to because of another.

Stéphane again came to the rescue with a patch, but because my system wasn’t booting I needed to go through the recovery process to install the update. Basically it just involves downloading the new IncusOS image onto a USB drive and rebooting the IncusOS box with the drive attached. You can see some specific instructions I followed in the related issue.

Fortunately the recovery process was very smooth and the system was updated and running quickly.