Qubes3.2 On MacBook Pro #qubes #shared

boot.sh

#!/bin/bash

# this script is currently manually run on boot

#stop on any error
set -e

#I use thunderbolt display, which doesn't hotplug correctly, so the ethernet on it can stick around
# remove it if it exists
qvm-pci -d sys-net 0d:00.0 || true

# My wifi card is 03:00.0, add it
sudo sh -c 'echo 03:00.0 > /sys/bus/pci/drivers/pciback/permissive'

# start sys-net without PCI
qvm-start sys-net

# add wifi via xl
xl pci-attach sys-net 03:00.0

# start the netvm service now (it is disabled on boot)
sudo systemctl start qubes-netvm

# if my thunderbolt is connected, attach the ethernet to sys-net also now
lspci | grep -q '0d:00.0' && qvm-pci -a sys-net 0d:00.0

# for good measure, send an update about monitor layouts
qubes-monitor-layout-notify

# Have had suspend issues with the Mac and the lid coming back on, disable things that wake it here
sudo sh -c 'echo LID0 > /proc/acpi/wakeup'
sudo sh -c 'echo XHC1 > /proc/acpi/wakeup'

Qubes3.2 On MacBook Pro.md

Qubes 3.2 on MacBook Pro Addendum

Notes based on this help doc: https://www.qubes-os.org/doc/macbook-troubleshooting/#macbook-pro-retina-15-inch-mid-2015-macbookpro-115

The above notes helped me 99% of the way there. Considering this is new territory for me though, that last 1% was rough and took learning a lot of things I was hoping to have been able to skip.

So here are some additional notes to that help doc...

1. Partioning

I didn't have any changes here, but I repartioned without rebooting in recovery mode using Disk Utility.

2. rEFInd

Having no clue what rEFInd was, nor EFI booting, nor how to possibly change your bootloader on a Mac, this was confisuing to me. I ended following the steps here for Elementary, which are the same for rEFInd: https://github.com/aroman/elementary-on-a-mac#install-refind-boot-manager

Basic steps were:

1. Download rEFInd ZIP file
2. Unzip it someplace
3. Reboot in recovery mode
4. Mount drive if needed from Disk Utility
5. Use Terminal to navigate to unzipped folder and run installer ./refind-install

I skipped the crsutil disable step that was in the Qubes doc (wasn't in the Elementary doc). Install was quick, and worked immediately on reboot.

3. Install

Steps listed were straightforward and worked great.

4. Grub Fix

Ignore the part about skipping this section, you need it later on.

1. chroot /mnt/sysimage
2. The second command here was hard to copy and pointed to the wrong file. Just edit /etc/grub.d/20_linux_xen and replace the single instances of multiboot with chainloader, module --nounzip with initrdefi, and module with linuxefi.
3. Skip the rest and go to step 5

5. Xen Fix

Edit /boot/efi/EFI/qubes/xen.cfg, the file will be empty. Use the following however, the notes had typoes and were missing the root which is a problem (including other stuff). This is taken from the UEFI Troubleshooting Doc

[global]
default=4.4.14-11.pvops.qubes.x86_64

[4.4.14-11.pvops.qubes.x86_64]
options=loglvl=all dom0_mem=min:1024M dom0_mem=max:4096M
kernel=vmlinuz-4.4.14-11.pvops.qubes.x86_64 root=/dev/mapper/qubes_dom0-root rd.lvm.lv=qubes_dom0/root rd.lvm.lv=qubes_dom0/swap i915.preliminary_hw_support=1 rhgb quiet
ramdisk=initramfs-4.4.14-11.pvops.qubes.x86_64.img

Use lsblk to find the right partion of your new Qubes install. For me, this was 5 on /dev/sda

Use the following to look back through cmds that were run that you will need to rerun.

grep Running /var/log/anaconda/program.log | tail -n 20

All of these cmds are based on the grep above:

1. Wipe out existing Qubes loader efibootmgr -b 0000 -B
2. Fix EFI location and use the correct partion number for your install: efibootmgr -c -w -L Qubes -d /dev/sda -p 5 -l \\EFI\\qubes\\xen-4.6.1.efi
3. Think we run this to be safe /usr/libexec/mactel-boot-setup
4. Regenerate grub config: grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg
5. Think this is ran to be safe too (takes a minute) kernel-install add 4.4.14-11.pvops.qubes.x86_64 /boot/vmlinuz-4.4.14-11.pvops.qubes.x86_64
6. This seems to have no effect, but I ran it anyway: systemctl disable qubes-netvm
7. This also breaks rEFInd being the bootloader, so you need to change the boot order to put the Mac being first so that rEFInd is used to start up. Use efibootmgr -v to see the details (and see refind is on the Mac) and then set the order to Mac then Qubes, for me this was efibootmgr -o 0080,0000
8. exit and reboot
9. You should be at rEFInd boot screen again, pick the EFI/qubes/xen-4.6.1.efi
10. Qubes installer should startup, use defaults and expect to crash during network setup because of wifi card.

6. Audio

I skipped this as I never had any audio issues.

7. Wifi Crash (disable wifi)

1. Force reboot and Hold Option after the chime
2. Pick your installed Qubes Partition to boot using grub
3. run configfile /EFI/qubes/grub.cfg
4. Once screen starts, hit e to edit config
5. Change set root=hd0;gpt4 to set root=hd1;gpt4
6. Hit Fn+CNTL+F2
7. Once you get back to setup screen hit Fn+CNTL+F4 and log in with your user account
8. Disable netvm sudo su - and systemctl disable qubes-netvm
9. Finish setup

8. Wifi fix

1. sys-net should not be running
2. launch termial in dom0
3. Increase swiotlb size qvm-prefs -s sys-net kernelopts "nopat iommu=soft swiotlb=8192"
4. Prep PCI card as described in steps

9. Suspend Issues

Good info here: https://joshtronic.com/2017/03/13/getting-suspend-in-linux-working-on-a-macbook-pro/ Basically, the mac would never go to sleep and burn the battery down because it was trying to wake back up. Those steps fixed it for me.

10. Resume Issues

WIFI card would stick freeze on resume. This was fixed by adding the wifi module to the suspend/resume load list on sys-net as described here: https://www.qubes-os.org/doc/wireless-troubleshooting/#automatically-reloading-drivers-on-suspendresume

Specifically, add brcmfmac to the /rm/config/suspend-module-blacklist on sys-net

11. Keyboard backtick fix

The MacbookPro backtick key was printing out < instead, needed to remap it. Found fix here: https://ubuntuforums.org/showthread.php?t=1486995&p=11297689#post11297689

12. Separate USB Qube

*NOTE: This is WIP and so far does not work at all... Basically Macbook has all USB (including mouse/keyboard) on a single USB controller. Have gotten around boot/login issues with the below steps, however simply assigning the controller to the USB-VM locks up (doesn't crash, just seems to lock up keyboard expect power button and trackpad). Note that this is still with allowing the qube policy of passing this through to dom0. I haven't dug too far into this yet, but it will eventually be a critical issue I'm guessing...

First enable sys-usb

sudo qubesctl top.enable qvm.sys-usb
sudo qubesctl state.highstate

Now, unhide it from dom0 so you can still boot up: sudo vi /boot/efi/EFI/qubes/xen.cfg And remove rd.hide.usb (or whatever it was called at the end of the kernel line)

Allow dom0 access to Keyboard and Mouse: edit /etc/qubes-rpc/policy/qubes.InputMouse and /etc/qubes-rpc/policy/qubes.InputKeyboard and add at the TOP: sys-usb dom0 allow,user=root

Finally, disable sys-usb autostart so you can still boot into dom0 for now. rm -f /etc/systemd/system/multi-user.target.wants/[email protected]

13. Wrap up with a boot.sh script

For now, I just run the attached boot.sh script when I boot up the machine. It handles everything for me. Eventually I'll move this to a systemd service...

Remaining Issues/Todo:

• Move boot.sh to be a systemd services, just haven't gotten around to it
• Annoying AppVm display resolution sometimes gets wonky. Haven't found root cause or fix. Killing VMs and running qubes-monitor-layout-notify repeatedly eventually solves it for me.
• Thunderbolt display hot plugging doesn't work, apparently Linux Kernel has resolved hot plugging for everything in thunderbolt Except for display (eg, ethernet works via thunderbolt hot-plug)
• No USB-VM means no proxying USB to VMs, only can do USB Drives for now
• Sometimes backtick keyboard change stops working in AppVMs and only works in dom0. I'm assuming this has to do with a suspend/resume issue, but it has only happened to me a couple times so far. Reboot fixes.