OpenWrt 23.05.x Setup Under LXD 5.x on Debian 12 with PCI Passthrough

In case you missed it, Canonical relicensed LXD under AGPLv3 in December 2023 with a mandatory CLA. The LXD project was hard forked as Incus and licensed under an Apache 2.0 License.

Incus is maintained by the same team of developers that first created LXD and is recommended for new users going forward.

If you continue with this guide using Incus, substitute incus for lxc in each command.

Install `lxd` via `snap` and configure

💡 If using Debian 12 (Bookworm) or newer, you can just apt install lxd but keep in mind this pacakge is EOL and new users are encouraged to move to Incus.

apt install snapd
snap install lxd

Configure `lxd`

The example config below uses the dir storage driver and no bridging since we'll be using PCI passthrough.

💡 If you have different needs, just run lxd init (or incus admin init) interactively and adjust accordingly.

lxd init --preseed << EOF
---
config: {}
networks: []
storage_pools:
- config: {}
  description: ""
  name: default
  driver: dir
profiles:
- config: {}
  description: ""
  devices:
    root:
      path: /
      pool: default
      type: disk
  name: default
projects: []
cluster: null
EOF

Set variables for convenience

💡 The current stable release of OpenWRT can be found here.

export SHORT_VERSION=23.05.2
export LONG_VERSION="openwrt-${SHORT_VERSION}-x86-64-generic-ext4-combined-efi"

Download EFI combined ext4 image and extract

💡 A decompression OK, trailing garbage ignored error is expected due to a harmless bug in the image builder.

wget "https://downloads.openwrt.org/releases/${SHORT_VERSION}/targets/x86/64/${LONG_VERSION}.img.gz"
gunzip "${LONG_VERSION}.img.gz"

Convert raw image to qcow2 image

💡 qemu-img can be installed via the qemu-utils package.

qemu-img convert -f raw -O qcow2 "${LONG_VERSION}.img" "${LONG_VERSION}.qcow2"

Create metadata file for image and compress

cat << EOF > metadata.yaml
---
architecture: x86_64
creation_date: $(date +%s)
properties:
    description: ${LONG_VERSION}
    os: openwrt
    release: ${SHORT_VERSION}
EOF
tar -cvzf metadata.tar.gz metadata.yaml

Import metadata and image to local lxc repository

lxc image import metadata.tar.gz "${LONG_VERSION}.qcow2" --alias "openwrt/${SHORT_VERSION}"

Create an new profile for the instance

This profile is configured for 4GiB of memory, 2 CPUs, 4GiB root disk, setting the instance to autostart on boot and disabling secureboot as the EFI image of OpenWrt we're using is unsigned. You will want to modify the values for parent devices to match your device names.

💡 For a full list of instance options, see here. If you're unsure what to set here, you can always change this later.

lxc profile create multi-wan
lxc profile edit multi-wan << EOF
---
config:
  limits.memory: 4GiB
  limits.cpu: 2,3
  boot.autostart: true
  security.secureboot: false
description: 3x NIC Passthrough
devices:
  eth0:
    name: eth0
    nictype: physical
    parent: eno2
    type: nic
  eth1:
    name: eth1
    nictype: physical
    parent: eno3
    type: nic
  eth2:
    name: eth2
    nictype: physical
    parent: eno4
    type: nic
  root:
    path: /
    pool: default
    type: disk
    size: 4GiB
name: multi-wan
used_by:
EOF

Initialize new instance derived from the `multi-wan` profile

💡 If you didn't define any limits.* above, you can pass --type and specify an AWS, GCE, or Azure instance type.

lxc launch --profile multi-wan --vm local:"openwrt/${SHORT_VERSION}" router

Connect to console

💡 Ctrl-a, q to escape

lxc console router

Expand rootfs

The disk image ${LONG_VERSION}.img is converted into the root disk during lxc launch. Usually this leaves a lot of free space on the disk (after the first partition) which can be expanded to occupy the full size of the disk. This is most easily accomplished with a short script from the OpenWrt Wiki.

💡 Read the contents of expand-root.sh before runnig this example: https://openwrt.org/docs/guide-user/advanced/expand_root

opkg update
opkg install parted losetup resize2fs

wget -O expand-root.sh "https://openwrt.org/_export/code/docs/guide-user/advanced/expand_root?codeblock=0"
source ./expand-root.sh

Update host machine networking defaults

Exclude your passthrough interfaces from being configured by the host machine. Multiple interfaces can be specified by separating them with a space inside the quotes.

/etc/default/networking

# Don't configure these interfaces. Shell wildcards supported.
EXCLUDE_INTERFACES="eno[234]"

Optional modifications

Reduce dhclient timers

If the host machine receives it's DHCP lease from its VM guest, you may also want to change dhclient's timeout and retry values so that the host machine doesn't enter into backoff before the guest is fully booted.

/etc/dhcp/dhclient.conf

timeout 30;
retry 30;

Review boot-related options

💡 You can edit your existing profile with lxc profile edit multi-wan

You may want to review the boot-related options in the LXD Documentation to control startup/shutdown and prioritization.

joshenders/openwrt_lxd_multiwan.md