A short guide on setting up Debian guests in bhyve

I have a problem. I need to run some Docker containers for work. Docker, as you may know, has a design that is tightly coupled to Linux. This is rather unfortunate since my desktop is running FreeBSD, and my laptop is running its distant cousin, macOS. Normally I'd just do such things on my Mac, but 1) my desktop has a lot more storage, and 2) the Docker images in question are amd64 already. Also I just wanted a challenge, so let's set up a VM to run Docker on FreeBSD.

Aside in case you're wondering why Docker runs on your Mac: Docker Desktop actually ships a hypervisor, HyperKit, to run Linux. If I'm not mistaken, this is actually based on a macOS port of bhyve. Read more here: https://medium.com/@hintcnuie/the-difference-between-xhyve-and-hyperkit-67a9f378ab97.

bhyve is a hypervisor for FreeBSD (and a few other platforms). There is no slick GUI or anything; it's all CLI-driven. This makes for a steep initial learning curve. Once you have everything set up though, it's great.

This is not an exhaustive guide, and I'm sure I missed a few things; please leave a comment if you notice anything off. The objective is to get a simple Debian base system installed with console access.

NOTE: All commands must be run as root.

Setting up bhyve

bhyve is part of the base system since FreeBSD 10.0. See the Handbook for a quick overview. All you need to do is make sure the kernel module is loaded. We will also install a helper package that makes things a little easier to work with.

kldload vmm
sysrc vmm_enable="YES"
pkg install vm-bhyve
sysrc vm_enable="YES"

Next, create a ZFS dataset where your VMs will live. My zpool is named zroot; change if yours is different.

zfs create zroot/vm
sysrc vm_dir="zfs:zroot/vm"

Finally, we can initialize things. I assume here that the zpool is mounted at /zroot, and that your network interface is named em0. This will initialize the bhyve manager, copy some predefined templates, and set up networking so your VM can talk to the net.

vm init
cp /usr/local/share/examples/vm-bhyve/* /zroot/vm/.templates/
vm switch create public
vm switch add public em0

Configuring the VM template

I personally like to codify settings in a template because I'm lazy and forgetful and this probably won't be the first time I have to spin up a VM. You could skip this step if you like and just edit the resulting config after vm create but I'll create a template here and talk through the settings. Here are the contents of mine, which lives at /zroot/vm/.templates/debian-zvol.conf.

loader="grub"
cpu=2
memory=16G
network0_type="virtio-net"
network0_switch="public"
disk0_name="disk0"
disk0_type="virtio-blk"
disk0_dev="sparse-zvol"
grub_run_partition="1"
grub_run_dir="/boot/grub"

Most of the lines are self-explanatory. Obviously, you probably don't need 16G of RAM unless you're doing something crazy with large amounts of data like I am.

Disk configuration

The disk settings are worth noting. I've set up a sparse zvol here. I won't elaborate too much on that, but zvols are a cool feature of ZFS. Some people say they aren't as performant as files; the FreeBSD handbook and several others (maybe copying it?) say the opposite. Do your own research. It works well enough and I got it working. You can find some more docs on disks in the vm-bhyve wiki.

Note that there isn't a way to specify the size. I tried a few things and always got a 25G disk. Someone leave a comment if you have a better idea. I ended up doing an ugly dance that we'll come back to in a moment.

Downloading the Installer ISO

Before installing, we need to download an ISO image and let our tool know about it. At the time of this writing, this is the URL for the latest Debian stable network installation image.

vm iso https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso

Creating the VM

This is pretty straightforward. We already configured a template above, so we just ask the manager to create a new VM with the template.

vm create -t debian-zvol debguest

At this point, the tool puts some files on disk in /zroot/vm/debguest, most notably a conf file which looks pretty similar to the template.

Starting the Install (sort of...)

Now you can kick off the installer!

vm install debguest debian-11.6.0-amd64-netinst.iso

This tells the manager to start the VM with our installer ISO as the boot device. Now that ugly thing about disks. If you run through the installer you'll realize that the disk has a size of 20G, and I couldn't find a great way to override this (comments welcome).

So after a few seconds I actually power off the vm. The volume does not appear to be created by this tool until the VM is actually powered on. I'm probably holding it wrong. Power it off, set the volume size to what you actually need, then restart the installer.

vm poweroff debguest
zfs set volsize=100G zroot/vm/debguest/disk0
vm install debguest debian-11.6.0-amd64-netinst.iso

The Console

Ok, so what happened? Well, nothing on your terminal. How do you connect to the dang thing?

vm console debguest

This script actually basically just wraps cu, a tool for directly connecting to other machines from a bygone era, but it actually still works. You might need to hit enter after connecting to wake it up or something.

Let me Out!!!

Exiting the console is even harder than exiting vi. How on earth to you get out? Well, you have to type ~.. Yes, ~. (or ~ followed by Ctrl-D). And it has to be at the start of a "line" to catch it, which means you may need to press enter before entering the magic instructions.

Wrapping up

That's about it! You now have a VM running Debian, and hopefully figured out how to enter and exit the console. Check out the manpage for vm to learn more about managing your VM (listing VMs, starting, etc.).

Hope this was helpful!

ianthetechie/bhyve-debian.md