Making Fusion-IO devices work on TrueNAS SCALE 22.02.1 and up

FusionIO on TrueNAS SCALE.md

FusionIO / Fusion-IO devices on TrueNAS SCALE

WARNING: This is unsupported and has potential to mess your system up. Proceed at your own risk.

Initial instructions were for 22.02.1 but also works for later versions with some additional steps. There's notes for each upgrade I've done at the end of the gist.

Note for v22.02.2:

• there's a bug where it doesnt show fusionio disks at all under Storage > Disks. So I had to downgrade to 22.02.1 and it works fine.

This gist is basically a note-to-self so future me can remember what I did to get this working on my TrueNAS server but I hope it helps someone else out too.

I wanted to use my iodrive duo (2 fusion devices on 1 card) as a mirrored SLOG on an existing raid2z pool.

Links to repos used:

• VSL3 driver: https://github.com/RemixVSL/iomemory-vsl
• VSL4 driver: https://github.com/RemixVSL/iomemory-vsl4
• FusionIO CLI Utilities/Tools: https://github.com/RemixVSL/vsl_downloads

Step 1) Install drivers

Note 1: (probably for v23.10.x and up)

If you get an error like zsh: command not found: make or something related to cc or dkms or dh_testdir missing, you'll need to run the following (as root user or using sudo) commands to install extra stuff then try again:

• chmod +x /bin/apt*
• chmod +x /usr/bin/dpkg*
• apt-get update
• apt-get install build-essential make dkms debhelper - press y to install when prompted

Note 2: (probably for v24.04.x and up)

If you get the message Package management tools are disabled on TrueNAS appliances. when you try to run sudo apt-get update, you can workaround it by running install-dev-tools then trying the commands from Note 1 again.

Note 3: (probably for v24.10.x and up)

Newer kernel with truenas' custom patches causes make dmks to fail. We'll have to use make dpkg instead.

Instructions

To download & install the VSL3 driver via dkms, run these commands in the shell (ssh or web gui):

• git clone https://github.com/RemixVSL/iomemory-vsl.git
• cd iomemory-vsl/
• for TrueNAS v24.04 and older: make dkms
• for TrueNAS v24.10 and newer: make dpkg
  • dpkg -i ../iomemory-vsl-$(uname -r)_*_$(dpkg --print-architecture).deb - installs the .deb file we just created

For VSL4:

• git clone https://github.com/RemixVSL/iomemory-vsl4.git
• cd iomemory-vsl/
• for TrueNAS v24.04 and older: make dkms
• for TrueNAS v24.10 and newer: make dpkg
  • dpkg -i ../iomemory-vsl-$(uname -r)_*_$(dpkg --print-architecture).deb - installs the .deb file we just created

Step 2) Install fio utilities/tools

IMPORTANT NOTE: you might need to unlock dpkg in order to use it for the last command: chmod +x /bin/dpkg*

• git clone https://github.com/RemixVSL/vsl_downloads.git
• cd vsl_downloads
• then look inside the vsl3 (or vsl4) directory for the .deb file.
• on ubuntu/debian linux version of truenas, install it by running: dpkg -i fio-util_3.2.16.1731-1.0_amd64.deb

Step 3) Reboot

After rebooting, you should now see the fusion cards present when running fio-status -a in the shell. If you need to upgrade firmware on your devices, find the instructions on how to generate the firmware over at vsl_downloads and use the fio-update-iodrive /dev/fctX path/to/firmware/file.fff command.

Step 4) Adding it to the zpool

If you don't see the drives, you need to wipe them with something like sgdisk --zap-all /dev/fioa (CAREFUL, you can easily wipe the wrong drive like this). Sometimes this also works for fio drives and old raid drives: wipefs -af /dev/fio{a,b} (again, be careful).

I can now see the drives in the GUI but when trying to add fioa and fiob drives as log devices to the pool, it would error out after trying to format the drives saying partition of type (zfs) couldn't be found. Weird thing is....the partitions are there (lsblk shows them). After this, the GUI would hang when trying to go to Storage > Disks to see the list of disks. Even after a reboot, nothing worked.

BUT after trying to manually add it to the zpool via the cli: zpool add [pool name] log mirror /dev/fioa /dev/fiob

AND then rebooting the server, it works and nothing hangs anymore! I can see the log drives in the GUI now + Storage > Disks doesn't hang anymore.

What happens after a version upgrade?

I'll share below what my upgrade experience looked like and list any issues or workarounds I needed

Upgrade from 22.02.4 to 22.12.4.2

• I did the upgrade through the web UI.
• Repeated steps 1-3 again.
  • Instead of the git clone ... command, run git pull once you're in the iomemory-vsl folder to get latest updates.
• I had to import the storage pool via the truenas GUI (for some reason running zpool import doesn't make the pool show up in the gui)

Upgrade from 22.02.4 to 23.10.2

• I did the upgrade through the web UI.
• Repeated steps 1-3 again.
  • Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
• Ran into issues with make dkms so I've updated the instructions in Step 1 with a workaround (thanks @TechnoidAlpha).
• After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

Upgrade from 23.10.2 to 24.04.2.5

• I did the upgrade through the web UI.
• Repeated steps 1-3 again.
  • Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
  • ran into this message Package management tools are disabled on TrueNAS appliances. when running sudo apt-get update. I've updated the instructions in Step 1 with a workaround (thanks again @TechnoidAlpha).
• After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

Upgrade from 24.04.2.5 to 24.10.0.2

• I did the upgrade through the web UI.
• Repeated steps 1-3 again.
  • Instead of the git clone ... command, run git pull once you're in the iomemory-vsl or vsl_downloads folder to get latest updates.
  • ran into errors when running make dkms. It was caused by the newer kernel versions. We gotta use make dpkg instead. I've updated the instructions in Step 1 with the workaround.
• After reboot, my existing pool showed up automatically and fio-status -a command shows my fusion drives!

Thanks for confirming this works on a new 24.10 install!

It's still odd that neither the steps or the .deb i sent above install for you.
What's the error you get when you try to install my .deb?

Confirmed, make dpkg works for 24.10

Thanks for confirming this works on a new 24.10 install!

It's still odd that neither the steps or the .deb i sent above install for you. What's the error you get when you try to install my .deb?

I think I'm on the same boat as @zinkscott . make dpkg refuses to build a .deb file successfully, and I cannot install the provided .deb file.
Also, I do confirm that spawning a fresh 24.10.0.2 installation works like a charm and builds the dpkg without following anything besides the steps you provided on Step1.

However, I'd like to have my current installation working as I have the feeling that down the road this might be useful.

dpkg -i with the .deb file provided by you @TheBoroer fails with the following error:

root@truenas-scale[/opt]# dpkg -i ./iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb
Selecting previously unselected package iomemory-vsl-6.6.44-production+truenas.
(Reading database ... 106980 files and directories currently installed.)
Preparing to unpack .../iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb ...
Unpacking iomemory-vsl-6.6.44-production+truenas (3.2.16.1732) over (3.2.16.1732) ...
Setting up iomemory-vsl-6.6.44-production+truenas (3.2.16.1732) ...
sudo: process 20738 unexpected status 0x57f
dpkg: error processing package iomemory-vsl-6.6.44-production+truenas (--install):
 installed iomemory-vsl-6.6.44-production+truenas package post-installation script subprocess was killed by signal (Killed)
Errors were encountered while processing:
 iomemory-vsl-6.6.44-production+truenas

I've traced the issue down to the dpkg process trying to execute the following control script

"runargv":["/var/lib/dpkg/info/iomemory-vsl-6.6.44-production+truenas.postinst","configure",""]

However, since the control script is missing the shebang, it'd fail miserably. I can replicate the issue as follows:

root@truenas-scale[/var/lib/dpkg/info]# ./iomemory-vsl-6.6.44-production+truenas.postinst
sudo: process 20821 unexpected status 0x57f
zsh: killed     ./iomemory-vsl-6.6.44-production+truenas.postinst

If I manually add the shebang it'd not complain and proceed, however I'm clueless on how to change the control script files inside your provided .deb, and why does it need to be changed, and not on other Scale 24.10.0.2 systems/VMs.

file contents for comparison

root@truenas-scale[/var/lib/dpkg/info]# cat iomemory-vsl-6.6.44-production+truenas.postinst
depmod 6.6.44-production+truenas

Hopefully this also helps finding a clue why make dpkg fails on this system as well.

Was able to alter and rebuild and install the .deb with the following:

mkdir temp_deb
cd temp_deb
ar p ../iomemory-vsl-debs/iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb control.tar.xz | tar -xJf -
ar p ../iomemory-vsl-debs/iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb data.tar.xz | tar -xJf -
echo "2.0" > debian-binary
nano postinst
nano postrm
nano prerm
tar -cJf control.tar.xz *
ar -r ../iomemory-vsl-modified.deb debian-binary control.tar.xz data.tar.xz
dpkg -i ../iomemory-vsl-modified.deb
cd ..
rm -rf temp_deb

Need to reboot still.

EDIT: Nanos were adding #!/bin/bash

It works!

The answer has been basically looking at me this whole time. In order to have make dpkg work, and to avoid adding the shebang (#!/bin/bash) manually in the deb packages scripts, you need to be logged in as root, not sudo'ing into root.

I've been trying to make dpkg and install the provided package logging into the system via SSH using my admin account, then sudo su. Even though you adquire root access, certain things seem to misbehave regarding the default shell interpreter (I suspect some environment variables do not get populated using this method) with the issues I've been suffering, and that pushed @zinkscott into manually unpacking and repacking @TheBoroer 's .deb file.

If you SSH into the system using your admin account, then su - root into "root" (requires to set a password for the root account using the UI, beforehand), you get all the environment variables (and God knows what else) but this allows to successfully compile the .deb package and/or directly install the provided .deb file by @TheBoroer

The reason why spinning a new, blank VM from scratch worked out for me, is that out of laziness I did not SSH into that VM, rather used the hypervisor's virtual console to issue all my commands, which logged me directly into root.

Hopefully this helps somebody in case they bump into an issue similar to mine. Thanks for all the support.

admin@truenas-scale[~]$ cat /etc/version
24.10.0.2%                                                                                                    
admin@truenas-scale[~]$ ls /dev/fct*
/dev/fct0  /dev/fct1
admin@truenas-scale[~]$

There was a change in 24.10 that removed the ability to sudo su -
https://ixsystems.atlassian.net/browse/NAS-131540
I ran su - and entered my admin password and was able to make module

make dpkg fails, but different:

Only thing that calls my attention is that somehow the dpkg-genbuildinfo stage wants to find a file that should be generated if you were running an old kernel version from TrueNAS 22.12.3 I believe. Perhaps delete the whole folder and re-cloning from source will help?

This is from a forum entry

You are literally two steps away from a successful compile.

Looking into iomemory-vsl_3.2.16.1732_amd64.buildinfo, seems like it contains checksums of all generated files. Perhaps deleting this file and have the make process rebuild it might help, just brainstorming here.

root@truenas-scale[/opt]# cat iomemory-vsl_3.2.16.1732_amd64.buildinfo | grep 6.6.44
Binary: iomemory-vsl-6.6.44-production+truenas iomemory-vsl-config-6.6.44-production+truenas iomemory-vsl-di-6.6.44-production+truenas iomemory-vsl-initrd-6.6.44-production+truenas iomemory-vsl-source
 1edd9e0e3542ad2621f9b592af0de81c 1346268 iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 5a48ec6e68c48702b936737353e6a5d8 11988 iomemory-vsl-config-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 3fb7412f4ab5771ddab9f68438e99e88 983468 iomemory-vsl-di-6.6.44-production+truenas_3.2.16.1732_amd64.udeb
 c7534d5a31b0e5f17ebaff707ae46033 9632 iomemory-vsl-initrd-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 35e2c1848c3bd15cf03ba0d3a9a9a48dbe97a317 1346268 iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 e1060373e981a808ac3f226cbe496748aae4c2d7 11988 iomemory-vsl-config-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 1bee66dfbcfc1906f4f2393e56b4e68b361ddf88 983468 iomemory-vsl-di-6.6.44-production+truenas_3.2.16.1732_amd64.udeb
 f5a504957cb80e6d9ba935a4ff7e1523cb9caba6 9632 iomemory-vsl-initrd-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 7b3e1d893c6a50b1a2d0a205ad05e06eca1a924246d8b317c687c7f260046573 1346268 iomemory-vsl-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 c44c54d673e82898abad6132ab0a63ac1388f3a16df492d859bb037ad5e0ed36 11988 iomemory-vsl-config-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 11e1e0808ab2c2bc91b9c685764ce5846191cb95d70ecbb828aa2c77f8da7fc5 983468 iomemory-vsl-di-6.6.44-production+truenas_3.2.16.1732_amd64.udeb
 c989639618ef91dc44d18851c3b560329e3b7cea26936d759f7679eaf743ff9b 9632 iomemory-vsl-initrd-6.6.44-production+truenas_3.2.16.1732_amd64.deb
 linux-headers-truenas-debug-amd64 (= 6.6.44-debug+truenas-1),
 linux-headers-truenas-production-amd64 (= 6.6.44-production+truenas-1),
 linux-truenas-production-libc-dev (= 6.6.44-production+truenas-1),

make dpkg method working in 24.10.1 as well

root@truenas-scale[~]# fio-status -a | grep fct
          fct0: Product Number:F00-001-1T20-CS-0001, SN:1228D2326
fct0    Attached
          fct1: Product Number:F00-001-1T20-CS-0001, SN:1206D3608
fct1    Attached
root@truenas-scale[~]# cat /etc/version
24.10.1
#