Ubuntu VM on macOS with QEMU

ubu-on-mac.md

Ubuntu VM on macOS with QEMU

For work I have an Intel Core i5 MacBook. With the Docker Desktop license changes I looked into minikube and Multipass but both failed when I got on our VPN with Cisco AnyConnect due to bridged networking in hyperkit (see links at bottom).

I realized I would be happy SSHing into a local Linux VM but passed on VirtualBox to avoid depending on legacy system extensions. I landed on libvirt for the VM, then realized I could remove libvirt from the equation to finally end up at this quick, simple setup for an Ubuntu VM on macOS with QEMU.

First, install QEMU (see https://wiki.qemu.org/Hosts/Mac for other options):

brew install qemu

Now create the file that will serve as the hard drive for the VM:

qemu-img create -f qcow2 ubuntu.qcow2 50g

Download an Ubuntu ISO file (or any other linux):

curl -LO https://releases.ubuntu.com/20.04/ubuntu-20.04.3-live-server-amd64.iso

Next, run this command to boot the VM (I put this in a script I call boot.sh):

qemu-system-x86_64 \
  -cdrom "ubuntu-20.04.3-live-server-amd64.iso" \
  -hda "ubuntu.qcow2" \
  -machine "type=q35,accel=hvf" \
  -m "4G" \
  -netdev "user,id=n1,hostfwd=tcp::2222-:22" \
  -device "virtio-net-pci,netdev=n1,bus=pcie.0,addr=0x19"

Explanation of options (many of these are lifted from https://www.arthurkoziel.com/qemu-ubuntu-20-04/):

• -cdrom boots the VM from the ISO file for installation. Drop it on subsequent boots once Ubuntu is installed.
• -hda specifies the hard drive file we'll be using.
• -machine uses the latest machine type and the macOS native hypervisor
• -m is the max amount of memory the machine may use
• -netdev and -device set up networking so we can ssh into the VM on port 2222

This command will open a QEMU window you can Alt-Tab to in order to monitor machine startup and proceed with installation. That window also has a handy "Power Down" dropdown option in the "Machine" item in the menu bar.

Once installation is done I add the contents of my public SSH key to ~/.ssh/authorized_keys in the VM, then ssh in with the following (starting or attaching to a screen session on login):

ssh -i ~/.ssh/gsf_id_ed25519 -p 2222 -t localhost 'screen -D -R'

I also need to share directories between host and guest, but Plan 9 and virtfs are not available in QEMU on macOS(see update below!), so I use sshfs (IP address of host is 10.0.2.2):

sshfs 10.0.2.2:/Users/gsf/Downloads ~/Downloads

That command will set that ~/Downloads mount until the next boot. To make it automatic, I generate an SSH keypair on the guest, add the public key to authorized_keys on the host, and make the sshfs mount on boot by adding the following to /etc/fstab:

sshfs#[email protected]:/Users/gsf/Downloads /home/gsf/Downloads fuse _netdev,allow_other,uid=1000,gid=1000,idmap=user,identityfile=/home/gsf/.ssh/id_ed25519 0 0

That's it! I kept notes on discoveries along the way below.

Notes

Multipass and minikube use hyperkit, which sets up bridged networking. This is what clashes with the AnyConnect client:

• canonical/multipass#961 (comment)
• https://multipass.run/docs/troubleshooting-networking-on-macos
• kubernetes/minikube#6296
• https://minikube.sigs.k8s.io/docs/handbook/vpn_and_proxy/

I'm still on an Intel MacBook but looking for a solution that I can continue using on an M1 processor. That rules out xhyve: machyve/xhyve#218

This was also helpful in laying out steps for running Ubuntu with QEMU on macOS: https://graspingtech.com/ubuntu-desktop-18.04-virtual-machine-macos-qemu/

Update! As of mid-March, homebrew's qemu supports 9p volumes. Following the qemu 9p setup page, I added this line to the qemu-system-x86_64 command for booting up the system:

    -virtfs local,path=/Users/gsf/Downloads,security_model=none,mount_tag=Downloads

And now my /etc/fstab line from above looks like so:

Downloads /home/gsf/Downloads 9p _netdev,trans=virtio,version=9p2000.u,msize=104857600 0 0

Note that I used version "9p2000.u", which differs from the "9p2000.L" suggested in the qemu 9p setup because I was getting a number of "Network dropped connection on reset" errors with the "L" version.

Because I went with security_model=none in the virtfs parameter, I had an issue where files in the share were owned by user ID 501 and group "dialout", which maps to the same group ID (20) as the "staff" group in macOS. I logged into the Ubuntu guest as root and ran the following to change the standard user and group IDs (both default to 1000):

# groupmod -g 120 dialout # move dialout out of the way
# groupmod -g 20 staff
# usermod -u 501 gsf
# usermod -g staff gsf
# chown -hR gsf:staff /home/gsf

I never had any issues with the sshfs volume, and given the mucking about in user and group IDs for 9p, I don't know if this is a step forward. I'll live with the new setup for a while and see.

@gsf Great work! Thanks for sharing!!

Just a few FYIs if anyone is interested:

• To know your own user id and info required at the folder mapping use the command id at a osx terminal.
• At least as today, brew seems out of date with the package for qemu. ports does have all this. To upgrade to the last version of qemu from ports use sudo port selfupdate && sudo port upgrade qemu.

@LRancez Thanks for that info on the MacPorts option!

I've switched back to sshfs for the moment as I was hitting some filesystem slowness with 9p (like 1 second pauses while editing a file in vim, and looong waits when grepping a directory). I may play with options like cache=mmap on the 9p mount to see if that helps.

Thanks for the instructions.

The file sharing with 9p does appear slow.

I am very new on Qemu. With VirtualBox, I used to set up host-only network for the VM and use nfs on it to share file between the host and the VM. Performance was good. host-only network is critical as at least the traffic doesn't go through the router, and it should also help with sshfs, and perhaps also 9p though I don't how it works. But I haven't figured out how to set up the network separately for Qemu on the Mac. Any hints?

I haven't tried anything other than a user network on qemu, but a search for qemu host-only network brings up a few good posts on the topic. The best I found was Configure NAT-network for QEMU on macOS Mojave. Unfortunately, and as noted in the comments on that post, that will not work on Big Sur and it seems TAP's days are numbered on MacOS.

When I tried to copy files from the guest fs to the 9p-mounted fs from the host, it said that there was not enough space available on the drive. I used the file manager GUI in LXDE on Debian Bullseye.