FusionIO / Fusion-IO devices on TrueNAS SCALE

WARNING: This is unsupported and has potential to mess your system up. Proceed at your own risk.

Initial instructions were for 22.02.1 but also works for later versions with some additional steps. There's notes for each upgrade I've done at the end of the gist.

Note for v22.02.2:

there's a bug where it doesnt show fusionio disks at all under Storage > Disks. So I had to downgrade to 22.02.1 and it works fine.

This gist is basically a note-to-self so future me can remember what I did to get this working on my TrueNAS server but I hope it helps someone else out too.

I wanted to use my iodrive duo (2 fusion devices on 1 card) as a mirrored SLOG on an existing raid2z pool.

Links to repos used:

VSL3 driver: https://github.com/RemixVSL/iomemory-vsl
VSL4 driver: https://github.com/RemixVSL/iomemory-vsl4
FusionIO CLI Utilities/Tools: https://github.com/RemixVSL/vsl_downloads

Step 1) Install drivers

Note 1: (probably for v23.10.x and up)

If you get an error like zsh: command not found: make or something related to cc or dkms or dh_testdir missing, you'll need to run the following (as root user or using sudo) commands to install extra stuff then try again:

chmod +x /bin/apt*
chmod +x /usr/bin/dpkg*
apt-get update
apt-get install build-essential make dkms debhelper - press y to install when prompted

Note 2: (probably for v24.04.x and up)

If you get the message Package management tools are disabled on TrueNAS appliances. when you try to run sudo apt-get update, you can workaround it by running install-dev-tools then trying the commands from Note 1 again.

Note 3: (probably for v24.10.x and up)

Newer kernel with truenas' custom patches causes make dmks to fail. We'll have to use make dpkg instead.

Instructions

To download & install the VSL3 driver via dkms, run these commands in the shell (ssh or web gui):

git clone https://github.com/RemixVSL/iomemory-vsl.git
cd iomemory-vsl/
for TrueNAS v24.04 and older: make dkms
for TrueNAS v24.10 and newer: make dpkg
- dpkg -i ../iomemory-vsl-$(uname -r)_*_$(dpkg --print-architecture).deb - installs the .deb file we just created

For VSL4:

git clone https://github.com/RemixVSL/iomemory-vsl4.git
cd iomemory-vsl/
for TrueNAS v24.04 and older: make dkms
for TrueNAS v24.10 and newer: make dpkg
- dpkg -i ../iomemory-vsl-$(uname -r)_*_$(dpkg --print-architecture).deb - installs the .deb file we just created

Step 2) Install fio utilities/tools

IMPORTANT NOTE: you might need to unlock dpkg in order to use it for the last command: chmod +x /bin/dpkg*

git clone https://github.com/RemixVSL/vsl_downloads.git
cd vsl_downloads
then look inside the vsl3 (or vsl4) directory for the .deb file.
on ubuntu/debian linux version of truenas, install it by running: dpkg -i fio-util_3.2.16.1731-1.0_amd64.deb

Step 3) Reboot

After rebooting, you should now see the fusion cards present when running fio-status -a in the shell. If you need to upgrade firmware on your devices, find the instructions on how to generate the firmware over at vsl_downloads and use the fio-update-iodrive /dev/fctX path/to/firmware/file.fff command.

Step 4) Adding it to the zpool

If you don't see the drives, you need to wipe them with something like sgdisk --zap-all /dev/fioa (CAREFUL, you can easily wipe the wrong drive like this). Sometimes this also works for fio drives and old raid drives: wipefs -af /dev/fio{a,b} (again, be careful).

I can now see the drives in the GUI but when trying to add fioa and fiob drives as log devices to the pool, it would error out after trying to format the drives saying partition of type (zfs) couldn't be found. Weird thing is....the partitions are there (lsblk shows them). After this, the GUI would hang when trying to go to Storage > Disks to see the list of disks. Even after a reboot, nothing worked.

BUT after trying to manually add it to the zpool via the cli: zpool add [pool name] log mirror /dev/fioa /dev/fiob

AND then rebooting the server, it works and nothing hangs anymore! I can see the log drives in the GUI now + Storage > Disks doesn't hang anymore.

What happens after a version upgrade?

I'll share below what my upgrade experience looked like and list any issues or workarounds I needed

Upgrade from 22.02.4 to 22.12.4.2

I did the upgrade through the web UI.
Repeated steps 1-3 again.
- Instead of the git clone ... command, run git pull once you're in the iomemory-vsl folder to get latest updates.
I had to import the storage pool via the truenas GUI (for some reason running zpool import doesn't make the pool show up in the gui)

Upgrade from 22.02.4 to 23.10.2

I did the upgrade through the web UI.
Repeated steps 1-3 again.
- Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
Ran into issues with make dkms so I've updated the instructions in Step 1 with a workaround (thanks @TechnoidAlpha).
After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

Upgrade from 23.10.2 to 24.04.2.5

I did the upgrade through the web UI.
Repeated steps 1-3 again.
- Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
- ran into this message Package management tools are disabled on TrueNAS appliances. when running sudo apt-get update. I've updated the instructions in Step 1 with a workaround (thanks again @TechnoidAlpha).
After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

Upgrade from 24.04.2.5 to 24.10.0.2

I did the upgrade through the web UI.
Repeated steps 1-3 again.
- Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
- ran into errors when running make dkms. It was caused by the newer kernel versions. We gotta use make dpkg instead. I've updated the instructions in Step 1 with the workaround.
After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

TheBoroer/FusionIO on TrueNAS SCALE.md