USB-key unlock on Ubuntu Server 22.04 LTS

uunlk.md

Unlock rootfs and Encrypted Drives with a USB Flash Drive [Linux/Ubuntu Server 22.04 LTS]

Setting up full disk encryption for Ubuntu is pretty straightforward. Enter all the neccessary info during instllation, and it magically works. But if you want to use a USB key not to enter the passphrase each and every time you boot your machine, as you would want to do with, say, a server, things quickly get really complicated.

But be not afraid, for you have found the right place to make it just a little bit easier. Join me, on this journey of frustration, bitter disappointment, and, as is always the case with linux, new knowledge.

Also, I should note, while using sudo -s generally is a bad idea, you might consider doing that, as pretty much all commands here require root access anyway. You'll just be typing five extra characters per command.

Setting up USB Unlock for rootfs

All we're essentially going to do to set up the USB key unlock, is make our system call a 'keyscript' every time our machine boots and use its output as the key. The executable responsible for calling the keyscript is apparently called cryptdisks_start. It reads the contents of /etc/crypttab and uses them to determine which disks need to be unlocked and how, then performs the neccessary actions: calls keyscripts, reads keys, calls cryptsetup, etc. We will thus need to edit /etc/crypttab to direct cryptdisks_start to call our keyscript and pass its output to cryptsetup.

By its design, cryptdisks_start will attach itself to the standard output of the keyscript and will use everything passed to it as the key. So if the script spits out the key and only the key, on standard output, then it's working correctly. If it instead spits out <some nonsense>+<the key> or <the key>+<some nonsense> or <the <some nonsense> key> or <nothing>, cryptsetup will error out with an ambiguos-as-heck message, so be sure to test your keyscript.

Before calling the script, cryptdisks_start will also set the CRYPTTAB_KEY environment variable to some or other nonsense of your choice, set in /etc/crypttab.

Here is the script that I use:

#!/bin/sh
set -e
if [ ! -e /key ]; then
  mkdir -p /key
  sleep 3
fi
for usbpartition in /dev/disk/by-id/usb-*; do
  usbdevice=$(readlink -f $usbpartition)
  if mount $usbdevice /key >/dev/null; then
    if [ -e /key/$CRYPTTAB_KEY.lek ]; then
      cat /key/$CRYPTTAB_KEY.lek
      umount $usbdevice
      exit
    fi
    umount $usbdevice
  fi
done
/lib/cryptsetup/askpass "Insert key or enter passphrase: "

As you can see, the script simply mounts each and every partition of each and every USB flash drive, checks if the file with the required key exists, and spits it out on standard output if it does. Otherwise, it asks for a passphrase. It doesn't have any filenames baked-in, but instead uses the CRYPTTAB_KEY environment variable. The value of this variable is, again, set by the options you use in /etc/crypttab, and is customizable for each and every encrypted drive, making this script very universal. Have multiple drives that need to be unlocked? Store multiple keys with different names on one flash drive, and use the same script for all drives.

Note that this script may not neccessarily work for you. I don't guarantee it. My USB key is formatted using ext4, and mount works fine with it with no options. If you have your flash drive formatted to FAT, you may want to add some options to mount. Test the script. See if it works, and proceed if it does.

Key Generation

The key to store on the flash drive can be generated using dd. But first I recommend you install uuid so you can generate random uuids and use them as names for your key files:

apt install uuid

Then just execute:

dd if=/dev/random bs=512 count=4 of=$(uuid).lek

It will generate some random nonsense into a .lek file. Put the .lek file in the root of your flash drive, and copy the name of that file somewhere. You will need it to set up /etc/crypttab.

Adding the Key to the Drive

Use lsblk to find the pet name of your encrypted partition. It's hard to miss, but if you need some clues as to where it is, it contains something of type 'crypt', which, in turn, contains the rootfs LVM volume. In my case, the partition was at /dev/sdb3. Using the partition pet name, you can then add the key you've just generated to that encrypted partition using

cryptsetup luksAddKey <the path to the partition> <the name of the .lek key file>

Here's an example of a correct command:

cryptsetup luksAddKey /dev/sdb3 11a84416-e040-11ec-be24-8f12a113b70f.lek

Modifying /etc/crypttab

The second to last step is to modify /etc/crypttab and put the script into the correct location.

Let's first address the script first. You need to put the script someplace where it will end up in your machine's initramfs, so there's a limited number of options where you can put it. I recommend the default location, which is /lib/cryptsetup/scripts as of writing this doc on Ubuntu Server 22.04 LTS. You can also RTFM to find the default location by typing man crypttab and searching for the keyscript option, it'll tell you where it is. Copy the script above into that directory and name it something like usb_unlock. Or come up with your own original name and replace usb_unlock with it whereever you see it. But don't forget to set it to be executable:

chmod a+x usb_unlock

If you don't make the script executable, update-initramfs will complain about keyscript location being invalid and nothing will work.

Now let's edit /etc/crypttab. Its format is relatively straightforward. Every line contains:

1. a target,
2. its corresponding partition-device-identificator-of-some-sorts,
3. the name of the key file
4. and additional options.

For me it looked kind of like this:

dm_crypt-0 UUID=<uuid of the encrypted drive (different from key UUID, unique for every partition, persists on reboot)> none luks

We will modify it to look like this:

dm_crypt-0 UUID=<uuid from before> <uuid of the key that we generated earlier, that is its filename except the .lek extension> luks,keyscript=usb_unlock,initramfs

Notice that none referred to the name of the key file, and needs to be gone and replaced with the correct UUID. This will be passed to the script as the value of the CRYPTTAB_KEY environment variable we discussed earlier. The keyscript option is also added, with a path to the script. The reason it's simply usb_unlock and not /usr/lib/cryptsetup/scripts/usb_unlock, is because we chose the default cryptsetup script directory, which just happens to be the one the path is interpreted relative to. If you wanted to put the script in, say, /bin, or /boot, you'd need to use the full path to the script, or make it relative to the aforementioned directory. The initramfs option is present to indicate that all of our keyscript routine needs to happen before the rootfs is mounted and thus needs to use the initramfs image. For some reason it worked for me without this option, but it seems that this is generally not the case, so consider playing with it, if you get issues.

Updating initramfs

After changing /etc/crypttab, you need to perform an initramfs update. The initramfs image is, as its name indicates, the file system available at boot, and cryptdisks_start needs it in order to be able to decrypt rootfs before it is loaded. So update it to include the usb_unlock script using

update-initramfs -u -k all

Pay close attention to the output of update-initramfs, as it will warn you if something's wrong with your crypttab configuration.

To check that the script is actually included in initramfs, use the lsinitramfs command like so:

lsinitramfs /boot/<initramfs image filename that update-initramfs tells you> | grep usb_unlock

If it outputs something with usb_unlock in it, highlighted in some color, then, congratulations, the script is in initramfs. If it doesn't, well, it's time to do some debugging!

I also did an update-grub for good measure after initramfs update, I don't know if this helps anything.

Adding More Drives

There are some scenarios, particularly in home use, where you will want to add more drives to the system. If you want to encrypt them as well, it's pretty easy to do so.

This time, we actually don't even need a keyscript, as cryptdisks_start appears to work its way through the /etc/crypttab entries sequentially, and by the time it requests the key for the Nth drive, where N>=2, the rootfs is already mounted. You can thus just put your keys into the /root directory (or pretty much any other directory, really) and use them as normal key files in /etc/crypttab without the whole keyscript thing. Just check that you're not leaking the keys in initramfs with

lsinitramfs /boot/<initramfs image filename that update-initramfs tells you> | grep lek

Preparing the Drive

To format the drive for encryption, generate a new key file for it, then use

cryptsetup --key-file <the name of the .lek key file> luksFormat /dev/<disk's pet name, get with lsblk, e.g. "sdc">

The drive should now be fully encrypted. This operation quite obviously also wipes any data from the drive, which it tells you it does. So be careful.

Now you can open the drive using

cryptsetup --key-file <the name of the .lek key file> luksOpen /dev/<pet name> <that random name that you assign, e.g. "<pet name>_crypt">

At this point articles suggest you do a

dd if=/dev/zero of=/dev/mapper/<that random name that you assign>

This command "zeroes" the drive fully (what it really does is writes zeroes to an unlocked drive, which makes the real data on the drive look like garbled encrypted mess from start to end) to "protect against disclosure of usage patterns", and can take anywhere from a few hours to eternity. But, honestly, even I am not paranoid enough to wait that long, so you do you on this one.

The drive is now unlocked, but it still doesn't have a filesystem on it. Let's fix it using

mkfs.ext4 /dev/mapper/<that random name that you assign>

If you want, you can now mount the drive. First of all, don't forget to create the directory to mount the drive to:

mkdir -p /mnt/<whatever>

Then actually mount it:

mount /dev/mapper/<that random name that you assign> /mnt/<whatever>

Modifying /etc/crypttab

This section differs slightly from what we did for rootfs. With rootfs, we assumed that /etc/crypttab was already pre-set-up for us by system installer. Thus, we didn't need to search for encrypted partition's UUID ourselves. This time, however, we'll need to use blkid. Its output, per line, looks a little bit like this:

/dev/<partition pet name>: UUID="<UUID of the drive partition as determined by the system, persists on reboot>" TYPE="<partition type>" <...>

Find the UUID of your drive's encrypted partition by the drive's pet name, and copy it. Pro tip: encrypted partitions have "crypto_LUKS" as their <partition type>. You will need the UUID to set up /etc/crypttab.

The entry we will add to /etc/crypttab will look like this:

<that random name that you assign> UUID=<UUID of the drive> <the directory where you put the .lek key file, e.g. "/root">/<the name of the .lek key file> luks

This is a bit too vague, too much stuff comes together in this line. But here's an example combining the "e.g."s from before:

sdc_crypt UUID=<UUID of the drive> /root/<the name of the .lek key file> luks

Still a bit too vague, but better. Anyway, go ahead, update initramfs and don't forget to put the .lek key file in the right directory, this is important!

Modifying /etc/fstab

The procedure for modifying /etc/fstab is relatively straightforward and easily found on the Internet, but here it is anyway. Just add a line like this one at the end:

/dev/mapper/<that random name that you assign> /mnt/<whatever> ext4 defaults 0 1

Just make sure that /mnt/<whatever> exists, by executing

mkdir -p /mnt/<whatever>

Or you can choose any other mount point you like.

Obviously, if you didn't format your drive as ext4, change fstab accordingly.

What to Do If You Messed Up

You may think that the most important point of this document is to teach you how to set up the USB key unlock. In fact, this is not the case. The most valuable piece of information you shall find here lies ahead. The 'How do you recover from "cryptsetup failed: bad password or options or moon phase or I dunno take a guess yourself" error' section.

In case anything goes wrong, which it will, you should wait for your system to error out. It will take anywhere from a few seconds to a few minutes, depending on how badly you screwed it up, but it will error out eventually. The system will drop you into an 'ash' shell, which runs on your system's initramfs image.

This is a useful place for a couple of reasons. For one, it allows you to boot into your encrypted rootfs with a few commands. But apart from that, it also allows you to test anything that you put in the initramfs image that cryptdisks_start needs to be able to access and/or execute.

Since you're already here, go ahead and test your keyscript. If it doesn't work (i.e. spits out anything other than the key), or worse, doesn't exist, then you'll have to fix it. Remember to set CRYPTTAB_KEY environment variable before testing the script, though. Think why it might not work, fix it in your actual rootfs where you have access to a text editor, and reboot.

The way you boot your rootfs is by using cryptsetup and manually entering the passphrase:

cryptsetup luksOpen /dev/<your encrypted partition (usually sda3 or sdb3 or whatever3)> <its target name in /etc/crypttab, like sda3_crypt or whatever3_crypt or dm_crypt-0>

An example of a correct command looks like this:

cryptsetup luksOpen /dev/sdb3 dm_crypt-0

If you don't know your encrypted partition's pet name, also known as what shows up when you do ls /dev, then you might want to type lsblk and search for something that has a 3 on the end, or otherwise looks suspiciously like an encrypted partition that would show up in /dev.

If you don't know what your partition is called in /etc/crypttab, go ahead and enter literally anithing instead. At this point, you might have an urge to hit your keayboard with a fist squarely in the middle. Go ahead and do that. Just remember that you will need your keyboard to proceed with the unlock.

I don't remember if the following command was neccessary to boot to rootfs, but I think that it was. Here it is:

vgchange -ay

You can do it after you've unlocked your drive with cryptsetup. This command activates LVM stuff or something. The point is, you won't be able to mount LVM volumes without this.

At this point, use exit to leave the limbo of initramfs. If you haven't messed up, it will just boot as if nothing happened. It will ask you the passphrase again though, if you used some random nonsense as your <its name in /etc/crypttab...>, so as soon as the system boots, open /etc/crypttab up, grab your phone, and take as many photos of your screen as you need to memorize the crypttab target name for your rootfs (aka the very first field of the entry). I should also note that update-initramfs will complain if you haven't used the correct name, and update-initramfs is probably not something you want to complain, so I would advise you to reboot, and enter the correct target name into cryptsetup command once you're in limbo.

All of the above is also applicable if you boot into a live environment from an installation USB stick, which you can use to modify rootfs without booting the system. Just remember that the Server version of the installation drive will never allow you to get rid of the installer, and you need to press Ctrl+Alt+F2 to get into the second shell instead. Once you're there, you can do pretty much anyhting you'd be able to do on an installed system, as far as I tested it.

First repeat all the steps neccessary to unlock the encrypted drive. Then, to mount rootfs in this environment, first use lvscan to see where your LVM volume went when you activated it, and then mount it, like so:

mount /dev/ubuntu-vg/ubuntu-lv /some-arbitrary-mountpoint

It is a good idea to chroot into it if you want to make modifications to it:

chroot /some-arbitrary-mountpoint

But update-initramfs still won't work correctly inside this chroot. At least it didn't work for me.

Thank you!
The secret to thorough and clear explanations is, every time I need to add a drive to a setup like this, I have to re-read this because I forget how it's done cuz I'm an idiot =P
Then I don't understand my own bad explanations and have to re-write it after I'm done.
The first draft was rough, you could feel how unmildly miffed I was writing all of this after two solid days of debugging XD
GitHub must still have it saved somewhere.
But I'm on my third drive added now, so we're getting there :)