Skip to content

Instantly share code, notes, and snippets.

@possibilities

possibilities/boxctl-provision-guide.md

Last active March 1, 2026 20:24
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save possibilities/89f2c2e118cda0827fe01055b69966cc to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save possibilities/89f2c2e118cda0827fe01055b69966cc to your computer and use it in GitHub Desktop.
Download ZIP
boxctl: Multi-Boot Reinstall + Provisioning Guide
Raw
boxctl-provision-guide.md

boxctl: Provision browserfarm -> artbird

Concrete guide for repurposing the browserfarm box. Keep the Trixie install (LUKS, p3), kill Bookworm (p4), provision with boxctl.

Current State

Hardware

  • CPU: Intel i7-9700 @ 3.00GHz (8 cores, no HT)
  • RAM: 32GB
  • LAN: 192.168.50.80
  • Tailscale: 100.91.28.68 (currently hostname browserfarm)

Disk Layout

nvme1n1 (465.8G)
├─ p1   512M  EFI (vfat)     /boot/efi   UUID=CFA4-67EF           SHARED
├─ p2   488M  ext2           /boot       UUID=36be5873-bdcf-...   TRIXIE /boot
├─ p3   360G  LUKS -> LVM   fatbird-vg  UUID=9263b308-939c-...   TRIXIE root (344G, 183G free)
└─ p4   105G  ext4           /           UUID=22c78c7b-314b-...   BOOKWORM (WIPE)

nvme0n1 (465.8G) Seagate BarraCuda - empty, no partitions
sda     (931.5G) SanDisk SSD - empty, no partitions

Trixie (p3) Inventory

Already installed: Debian 13.1, kernel 6.12.48+deb13-amd64, Chrome 140, Xorg + dummy driver, htop, Tailscale 1.94.2

Hostname: blackbird Users: mike (93GB home), root -- no artbird yet SSH key deployed: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGgDW/IiZM5KhlUPbLkrsdVOW4qAG/5Df6pdp3j9B00C

/opt contents (old): containerd, google (chrome), Min, Telegram, Telegrxam, tuui, zig

Missing for boxctl: artbird user, SSH hardening, restic, tmux, rsync, jq, curl, sudo, backup.env, iptables ARTBIRD chain, route_localnet, python venv, boxctl/browserctl CLIs, fvwm3, tigervnc, browserctl config + systemd unit

Boot Situation

  • Bookworm (p4) currently owns EFI GRUB (Boot0002/0009 -> \EFI\DEBIAN\)
  • Trixie has its own GRUB in p2 /boot/grub/ but Bookworm chainloads to it via os-prober
  • Stale EFI entries: Windows Boot Manager (Boot0000), ubuntu (Boot0001) -- no matching partitions

Known Issues to Fix

  1. crypttab says nvme0n1p3_crypt but LUKS is on /dev/nvme1n1p3 -- wrong device name, boot will fail standalone
  2. fstab swap entry (fatbird--vg-swap_1) -- may not exist as an LV, needs verification
  3. GRUB_DISABLE_OS_PROBER commented out (defaults to true on Trixie) -- needs enabling
  4. PV size mismatch on fatbird-vg -- pvresize needed

Phase 0: Backup Browser State

Before touching anything, pull the browser profiles and config off the live box to ~/Desktop on the Mac. This preserves all logged-in sessions (cookies, local storage, service workers) so they survive the reinstall.

What gets backed up

~/Desktop/browserfarm-backup/
  base-profile/          # Auth state: cookies, login data, local storage, service workers
  config/                # /etc/browserctl/ (config.toml, backup.env, etc.)

The base-profile contains everything needed for logged-in sessions:

Default/Cookies, Cookies-journal
Default/Preferences, Secure Preferences
Default/Login Data, Login Data-journal
Default/Web Data, Web Data-journal
Default/Local Storage/
Default/Session Storage/
Default/IndexedDB/
Default/Service Worker/
Default/Extension State/
Local State
First Run

Step 0a: Record Chrome version

Note the Chrome version on the server for profile compatibility. If the new install ships a much newer Chrome, profiles may get auto-migrated (one-way).

ssh browserfarm@browserfarm "google-chrome --version"

Step 0b: Pull browser profiles

mkdir -p ~/Desktop/browserfarm-backup

# Base profile (auth state, cookies, local storage -- the important stuff)
rsync -az browserfarm@browserfarm:/var/lib/browserctl/base-profile/ \
  ~/Desktop/browserfarm-backup/base-profile/

# browserctl config (config.toml, backup.env, backup-exclude.txt, sites.yaml)
rsync -az browserfarm@browserfarm:/etc/browserctl/ \
  ~/Desktop/browserfarm-backup/config/

Step 0c: Verify the backup

# Check that critical profile files exist
ls ~/Desktop/browserfarm-backup/base-profile/Default/Cookies
ls ~/Desktop/browserfarm-backup/base-profile/Default/Preferences
ls ~/Desktop/browserfarm-backup/base-profile/Default/Login\ Data
ls ~/Desktop/browserfarm-backup/base-profile/Default/Local\ Storage/

# Check config
ls ~/Desktop/browserfarm-backup/config/

If base-profile/ is empty or missing Default/Cookies, the daemon may not have saved profiles yet -- check with ssh browserfarm@browserfarm "ls -la /var/lib/browserctl/base-profile/".

Phase 1: Make Trixie Boot Standalone

Step 1: Boot into Trixie

Reboot the box. At the GRUB menu, select "Debian GNU/Linux 13 (trixie) (on /dev/mapper/fatbird--vg-root)". Enter the LUKS passphrase when prompted.

Step 2: Fix crypttab

The crypttab references the wrong device name. Fix it:

# Check current (broken)
cat /etc/crypttab
# nvme0n1p3_crypt UUID=9263b308-939c-4744-889a-fc11705a5959 none luks,discard

# Verify the UUID matches nvme1n1p3
sudo blkid /dev/nvme1n1p3
# Should show UUID="9263b308-939c-4744-889a-fc11705a5959"

# Fix the device name (the UUID is correct, but rename for clarity)
sudo sed -i 's/nvme0n1p3_crypt/nvme1n1p3_crypt/' /etc/crypttab
cat /etc/crypttab
# nvme1n1p3_crypt UUID=9263b308-939c-4744-889a-fc11705a5959 none luks,discard

The UUID-based lookup means it'll find the right device either way, but the name should match reality.

Step 3: Fix swap LV

# Check if the swap LV exists
sudo lvs
# If fatbird--vg-swap_1 is NOT listed, comment it out of fstab:
sudo sed -i 's|^/dev/mapper/fatbird--vg-swap_1|#/dev/mapper/fatbird--vg-swap_1|' /etc/fstab

# If it IS listed but inactive:
sudo lvchange -ay fatbird-vg/swap_1
sudo swapon /dev/mapper/fatbird--vg-swap_1

Step 4: Fix PV size mismatch

sudo pvresize /dev/mapper/nvme1n1p3_crypt
sudo pvs  # Should show correct size with no warnings

Step 5: Regenerate initramfs

After crypttab changes, rebuild initramfs so the boot process uses the corrected config:

sudo update-initramfs -u -k all

Step 6: Take over GRUB

# Install GRUB for UEFI
sudo apt-get update
sudo apt-get install -y grub-efi-amd64 os-prober

# Enable os-prober (Trixie disables it by default)
echo 'GRUB_DISABLE_OS_PROBER=false' | sudo tee -a /etc/default/grub

# Install GRUB to the EFI partition
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian

# Regenerate GRUB config
sudo update-grub

Step 7: Reboot and verify

sudo reboot

At the GRUB menu, Trixie should now be the primary entry. Boot into it and confirm everything works. You should be able to SSH in as mike@browserfarm (Tailscale hostname hasn't changed yet).

If it doesn't boot: Use a live USB, mount the LUKS + LVM, chroot in, and fix. The Bookworm install on p4 is still there as a fallback at this point.

Phase 2: Wipe Bookworm

Only do this after Phase 1 is confirmed working.

Step 8: Wipe the Bookworm partition

# From inside Trixie
sudo mkfs.ext4 -L artbird-spare /dev/nvme1n1p4

Step 9: Clean up GRUB

sudo update-grub
# The stale Bookworm entry should disappear

Step 10: Clean stale EFI entries

# List entries
sudo efibootmgr -v

# Remove stale Windows and Ubuntu entries (adjust numbers as needed)
sudo efibootmgr -b 0000 -B   # Windows Boot Manager
sudo efibootmgr -b 0001 -B   # ubuntu

# Verify
sudo efibootmgr -v

Phase 3: Provision with boxctl

Now you're on a clean Trixie with full control. Run everything as root.

Credentials

Have these ready:

export TAILSCALE_AUTH_KEY="tskey-auth-..."
export B2_KEY_ID="..."
export B2_APP_KEY="..."
export B2_BUCKET="..."
export RESTIC_PASSWORD="..."

Layer 1: Base System

# 1. Change hostname
echo "artbird" > /etc/hostname
sed -i 's/blackbird/artbird/g' /etc/hosts
hostname artbird

# 2. Create artbird user
useradd -m -s /bin/bash -G video,audio,sudo artbird
passwd artbird

# 3. SSH key + hardening
mkdir -p /home/artbird/.ssh
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGgDW/IiZM5KhlUPbLkrsdVOW4qAG/5Df6pdp3j9B00C notimpossiblemike@gmail.com" > /home/artbird/.ssh/authorized_keys
chown -R artbird:artbird /home/artbird/.ssh
chmod 700 /home/artbird/.ssh && chmod 600 /home/artbird/.ssh/authorized_keys

cat > /etc/ssh/sshd_config.d/artbird.conf << 'EOF'
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM no
EOF
systemctl restart sshd

# 4. Tailscale -- re-register as artbird
#    First: remove old "browserfarm" node from Tailscale admin console
#    Then:
tailscale down
tailscale up --authkey "$TAILSCALE_AUTH_KEY" --hostname artbird --reset

# 5. Packages (Chrome, Xorg, htop already installed)
apt-get update
apt-get install -y rsync jq curl sudo tmux restic python3 python3-pip python3-venv

# 6. Backup
mkdir -p /etc/artbird
cat > /etc/artbird/backup.env << EOF
B2_ACCOUNT_ID=$B2_KEY_ID
B2_ACCOUNT_KEY=$B2_APP_KEY
B2_BUCKET=$B2_BUCKET
RESTIC_PASSWORD=$RESTIC_PASSWORD
RESTIC_REPOSITORY=b2:$B2_BUCKET:artbird
EOF
chmod 600 /etc/artbird/backup.env
source /etc/artbird/backup.env && restic init

# 7. iptables
iptables -t nat -N ARTBIRD
iptables -t nat -A PREROUTING -j ARTBIRD
TS_IFACE=$(ip -o link show | grep tailscale | awk -F: '{print $2}' | tr -d ' ' | cut -d@ -f1)
sysctl -w net.ipv4.conf.$TS_IFACE.route_localnet=1
echo "net.ipv4.conf.$TS_IFACE.route_localnet = 1" > /etc/sysctl.d/artbird.conf

# 8. Python venv
python3 -m venv /opt/browserctl-venv

From Your Mac: Push CLIs

# After Tailscale re-registers, verify connectivity
tailscale ping artbird

# Push the code
rsync -az ~/code/arthack/apps/ artbird@artbird:/opt/arthack/apps/

# Install CLIs
ssh artbird@artbird "source /opt/browserctl-venv/bin/activate && pip install /opt/arthack/apps/cli_common /opt/arthack/apps/boxctl /opt/arthack/apps/browserctl"

Layer 2: Browser Service

# Xorg dummy driver config (xorg already installed)
mkdir -p /etc/X11/xorg.conf.d
cat > /etc/X11/xorg.conf.d/10-dummy.conf << 'EOF'
Section "Device"
    Identifier "DummyDevice"
    Driver "dummy"
    VideoRam 256000
EndSection
Section "Screen"
    Identifier "DummyScreen"
    Device "DummyDevice"
    Monitor "DummyMonitor"
    DefaultDepth 24
    SubSection "Display"
        Depth 24
        Modes "1920x1080"
    EndSubSection
EndSection
Section "Monitor"
    Identifier "DummyMonitor"
    HorizSync 28.0-80.0
    VertRefresh 48.0-75.0
EndSection
EOF

# Window manager + VNC (Chrome already installed)
apt-get install -y fvwm3 tigervnc-standalone-server tigervnc-common

# Config + directories
mkdir -p /etc/browserctl /var/lib/browserctl /tmp/browserctl/pool
chown -R artbird:artbird /var/lib/browserctl /tmp/browserctl
cat > /etc/browserctl/config.toml << 'EOF'
[service]
display = ":0"
pool_dir = "/tmp/browserctl/pool"
data_dir = "/var/lib/browserctl"

[chrome]
binary = "/usr/bin/google-chrome-stable"

[vnc]
enabled = true
EOF

# Systemd service
cat > /etc/systemd/system/browserctl.service << 'EOF'
[Unit]
Description=browserctl browser service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=artbird
Environment=DISPLAY=:0
ExecStart=/opt/browserctl-venv/bin/browserctl run-service
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable browserctl && systemctl start browserctl

Cleanup (optional)

# Remove old /opt junk from previous life
rm -rf /opt/containerd /opt/Min /opt/Telegram /opt/Telegrxam /opt/tuui /opt/zig

# Clean up mike's 93GB home if no longer needed
# userdel -r mike  # only after confirming nothing is needed

Phase 4: Restore Browser State

After provisioning completes and services are running, push the backed-up profiles back to the new box. This restores all logged-in sessions from the old browserfarm install.

Step 11: Push profiles to artbird

# Restore base profile (auth state, cookies, local storage)
rsync -az ~/Desktop/browserfarm-backup/base-profile/ \
  artbird@artbird:/var/lib/browserctl/base-profile/

# Fix ownership (rsync preserves Mac uid/gid which won't match)
ssh artbird@artbird "sudo chown -R artbird:artbird /var/lib/browserctl/base-profile"

Step 12: Restore browserctl config (selective)

Don't blindly overwrite the fresh config -- the new box may have different paths or settings. Cherry-pick what's useful:

# Compare old vs new config
diff ~/Desktop/browserfarm-backup/config/config.toml <(ssh artbird@artbird "cat /etc/browserctl/config.toml")

# If the old config has sites.yaml, push it
if [ -f ~/Desktop/browserfarm-backup/config/sites.yaml ]; then
  rsync -az ~/Desktop/browserfarm-backup/config/sites.yaml \
    artbird@artbird:/etc/browserctl/sites.yaml
fi

Step 13: Bounce the daemon and verify sessions

# Restart browserctl so it picks up the restored profiles
ssh artbird@artbird "sudo systemctl restart browserctl"

# Wait for pool to come up, then validate sessions
browserctl ensure-sessions /etc/browserctl/sites.yaml

If any sessions are stale (cookies expired, tokens rotated), use VNC to re-authenticate manually:

  • Connect to artbird:5910 (slot 0) via a VNC client
  • Acquire a browser, navigate to the stale site, log in
  • Save the profile: browserctl save-profile <id> --session manual-fix

Verification

# SSH in as artbird
ssh artbird@artbird

source /opt/browserctl-venv/bin/activate
boxctl check-system        # All 11 checks green
browserctl check-service   # Browser stack healthy

boxctl check-system expects

Check Expected
OS: Debian Trixie debian trixie
User: artbird (video, audio, sudo) ok
Tailscale: installed, connected, hostname=artbird connected, hostname=artbird
SSH: key-only auth key-only
Backup: restic + backup.env ok
iptables: ARTBIRD chain chain exists
route_localnet enabled tailscale0: enabled
Core packages all present
Disk: 5+ GB free ~183 GB free
RAM: 2+ GB available ~28 GB available
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment