zvol on optane for a swap device

Using a zvol for a swap device for host and VMs



Swapping to Optane

Since i’m running virtual machines, there is another thing which should go to low-latency storage: swap. I try to conserve as much memory as possible, which means VMs sometimes use their swap space, which gets horribly slow in case it resides on spinning disks. For that reason i created another partition, created a separate ZFS pool and created disk images that will hold the VMs swap.

Creating a new pool is very simple and as i don’t need redundancy on swap it will just be one “device”, actually a partition. Using unique hardware identifiers instead of device paths (e.g. “/dev/nvme0n1p3”) is quite helpful as PCIe enumeration and partition order may change.

$ zpool create \ 
    -O normalization=formD \ 
    -O sync=always \
    SWPS INTEL_SSDPED1D280GA_PHMXXX2301DU280CGN-part2
    
Now create a zvol as a swap device but you'll need to configure it appropriately.

- Set the volume block size to match your systems page size. This tuning prevents ZFS from having to perform read-modify-write options on a larger block while the system is already low on memory.
- Set the logbias=throughput and sync=always properties. Data written to the volume will be flushed immediately to disk freeing up memory as quickly as possible.
- Set primarycache=metadata to avoid keeping swap data in RAM via the ARC.
- Disable automatic snapshots of the swap device.
( 4GB works for me but size accordingly.  I'm not sure if this duplication of settings is needed, i.e. sync=always, as the zvool should inheret these from the pool.  In which case maybe all the options should go to the pool?)


$ zfs create -V 4G -b $(getconf PAGESIZE) \
    -o logbias=throughput -o sync=always \
    -o primarycache=metadata \
    -o com.sun:auto-snapshot=false \
    SWPS/swap


Now new virtual disks are created on this ZFS pool and get attached to their virtual machine.	

$ zfs list
swaps                  33.1M  96.8G    24K  /swaps
swaps/vm-100-disk-1      30K  96.8G    30K  -
swaps/vm-101-disk-1    1.02M  96.8G  1.02M  -
...

Replacing old swap and re-claiming that space for the root partition is easy if the VMs are using LVM. /dev/sdb is the new virtual device available to the VM, stored at the ZFS “swaps” pool on Optane.

Add the new swap space to LVM:

$ pvcreate /dev/sdb
$ vgcreate swp /dev/sdb
$ lvcreate -l 100%FREE -n swap swp

Create the swap file system and use the UUID as device identifier in /etc/fstab:

$ mkswap /dev/vm-optane/swap 
$ vim /etc/fstab
/dev/mapper/swp-swap none swap sw 0 0

Disable and remove the old swap partition:	

$ swapoff /dev/vm-system/swap 
$ lvremove /dev/vm-system/swap

Extend the root partition and file system to use the free’d up space:	

$ lvextend -l +100%FREE /dev/vm-system/root
$ resize2fs /dev/vm-system/root

…and reboot the VM, just to be sure the file system is undamaged.