dakota incus

incus-plan-sysext.md

Here’s the concrete sysext plan I’d use.

Goal Build an incus.raw system extension that overlays only /usr, while keeping Incus state on the host in /var/lib/incus, /var/cache/incus, /var/log/incus, and /run/incus.

1. Define The Artifact Create a separate dakota-incus-sysext build artifact, initially out-of-tree.

The sysext should contain:

/usr/bin/incus
/usr/bin/incusd
/usr/bin/incus-user
/usr/bin/incus-migrate
/usr/bin/incus-agent
/usr/lib/incus/...
/usr/lib/systemd/system/incus.service
/usr/lib/systemd/system/incus-user.service
/usr/lib/systemd/system/incus-user.socket
/usr/lib/systemd/system/lxcfs.service
/usr/lib/systemd/system-preset/80-incus.preset
/usr/lib/sysusers.d/incus.conf
/usr/lib/tmpfiles.d/incus.conf
/usr/lib/extension-release.d/extension-release.incus
/usr/share/bash-completion/completions/incus
/usr/share/zsh/site-functions/_incus
/usr/share/fish/vendor_completions.d/incus.fish

The extension release file must match Dakota. Today that means something like:

ID=bluefin-dakota
VERSION_ID=20260621

But that is brittle for a rolling image. Better Dakota-side fix: add a stable SYSEXT_LEVEL=dakota-1 to the base image, then build sysexts against that instead of daily VERSION_ID.

2. Package Required Runtime Dependencies Minimum container support payload:

incus
lxc >= 6.0
lxcfs
cowsql / raft
libacl
libcap
libuv
sqlite
seccomp
nftables
dnsmasq
skopeo
rsync
tar / xz / gzip / zstd helpers as needed

Storage backends:

dir: no extra package, good default
btrfs: needs btrfs-progs if enabled
lvm: needs lvm2 + thin provisioning tools
zfs: defer initially unless Dakota kernel/module story is clear
ceph: defer

I’d ship dir + btrfs + lvm first, but default initial setup to dir.

3. Add VM Support For Incus VMs, the sysext must include:

qemu-system-x86_64
qemu-img
qemu-storage-daemon if required by packaged QEMU split
edk2/OVMF firmware
swtpm
swtpm_setup
virtiofsd, preferably the Rust virtiofsd
seabios only if legacy BIOS support matters
dnsmasq
nftables

Host requirements to validate:

/dev/kvm exists
kvm kernel module loaded
user can access /dev/kvm via group or udev rule
cgroup v2 mounted
tun/tap available
vhost_net available if using virtio networking

Add sysext udev/rules if needed:

/usr/lib/udev/rules.d/60-incus-kvm.rules

But be conservative here; if Dakota already handles kvm group/device permissions, reuse that.

4. Systemd Units Ship native host units from the sysext.

incus.service:

[Unit]
Description=Incus daemon
After=network-online.target lxcfs.service
Wants=network-online.target lxcfs.service

[Service]
Type=simple
ExecStart=/usr/bin/incusd --group incus-admin
KillMode=process
TimeoutStartSec=600s
TimeoutStopSec=30s
Restart=on-failure
LimitNOFILE=1048576
TasksMax=infinity

[Install]
WantedBy=multi-user.target

lxcfs.service should mount/run LXCFS before Incus. The exact command depends on how LXCFS is packaged, but it must expose the usual container-aware proc/cgroup views.

Preset:

enable lxcfs.service
enable incus.service
enable incus-user.socket

5. Host-Writable Setup Helper Because sysext cannot write /etc, add a helper in the sysext:

/usr/libexec/incus-dakota-setup
/usr/bin/setup-incus-sysext

It should idempotently do:

groupadd --system incus-admin if missing
groupadd --system lxc if needed
ensure root has /etc/subuid range
ensure root has /etc/subgid range
systemd-tmpfiles --create incus.conf
systemctl daemon-reload
systemctl enable --now lxcfs.service incus.service incus-user.socket

Also wire Dakota’s existing ujust dx-group path so the user gets added to incus-admin. That recipe already knows about incus-admin.

6. First Initialization Do not auto-create a bridge/storage pool silently. Provide a ujust setup-incus wrapper:

sudo setup-incus-sysext
sudo incus admin init

Offer a non-interactive low-risk default:

config: {}
networks: []
storage_pools:
- name: default
  driver: dir
profiles:
- name: default
  devices:
    root:
      path: /
      pool: default
      type: disk
projects: []
cluster: null

Networking can be a second explicit step because bridges/firewall rules are more invasive.

7. Build Pipeline Use a containerized builder, not the host.

Pipeline shape:

build rootfs matching Dakota ABI
build/package LXC, LXCFS, Incus, QEMU stack
stage files under rootfs/usr
add extension-release metadata
create erofs or squashfs image
sign/checksum artifact
publish incus.raw

I’d start with a plain mkosi or systemd-repart based sysext builder, then later migrate the recipe into Dakota BuildStream if it proves useful.

8. Install Flow Manual install prototype:

sudo mkdir -p /var/lib/extensions
sudo cp incus.raw /var/lib/extensions/incus.raw
sudo systemctl enable --now systemd-sysext.service
sudo systemd-sysext refresh
sudo setup-incus-sysext
incus admin init

Validation:

systemd-sysext status
incus version
systemctl status incus lxcfs
incus info
incus launch images:debian/13 c1
incus launch images:debian/13 vm1 --vm

9. Risks Biggest risks:

• ABI drift between Dakota and the sysext.
• QEMU/firmware size and dependency sprawl.
• /etc/subuid and /etc/subgid cannot be owned by sysext.
• AppArmor vs SELinux policy needs investigation on Dakota.
• ZFS should probably wait until kernel/module handling is understood.

My practical first milestone would be: containers only, dir storage, no bridge auto-init. Second milestone: VMs with QEMU + OVMF + swtpm + virtiofsd. Third milestone: decide whether this graduates from external sysext into Dakota BuildStream image content.