TrueNas nvidia grid drivers setup

nvidiagrid.md

WARNING: What I did is not officially supported, and I’m far from an expert. You risk breaking your system by following these steps. Proceed with caution. If you plan to try this yourself, make sure to back up your configuration first, and consider testing everything in a temporary VM before applying it to your main system.

1. Disable nvidia drivers from the app settings
   • Go to: configuration->settings
   • uncheck “install nvidia drivers”)
   • reboot your system
2. Enable root login by going to credentials->users->root and add a password to the root user
3. Become root by running su -
4. Enable developer mode by running install-dev-tools (see: Developer Mode (Unsupported) | TrueNAS Documentation Hub)
5. Download the NVIDIA GRID drivers that you need into a pool of your choice, otherwise they won't be runnable. I followed this guide and downloaded the drivers from the google compute website as described.
   • In my case the complete command was wget -O nvidia.run https://storage.googleapis.com/nvidia-drivers-us-public/GRID/vGPU16.4/NVIDIA-Linux-x86_64-535.161.07-grid.run -P /mnt/main
6. Install the drivers:
   • CD in the pool that you choose to use
   • Create a temporary directory by running mkdir test
   • Make the file executable by running chmod +x nvidia.run
   • Run the installer with the --tmpdir flag pointing to that directory.
   • Example (for NVIDIA 535 drivers with a pool called main): ./nvidia.run --dkms --tmpdir /mnt/main/test
7. Install the nvidia container toolkit by following the official guide

Important notes:

• After every reboot, your GPU UUID will change. You will have to update the assigned GPU UUID for your app.
• To find your gpu's uuid run midclt call app.gpu_choices | jq
• To change the uuid assigned to your app run midclt call -job app.update APP_NAME '{"values": {"resources": {"gpus": {"use_all_gpus": false, "nvidia_gpu_selection": {"PCI_SLOT": {"use_gpu": true, "uuid": "GPU_UUID"}}}}}}' after replacing the following:

• APP_NAME is the name you entered in the application, for example “plex”.
• PCI_SLOT is the pci slot identified in the error that you will get when trying to run the app before having changed the uuid, for example "0000:2d:00.0”.
• GPU_UUID is the UUID matching the pci slot that you retrieved with the above command.

as described here.

I see the difference from what i was doing.

I never did step 2 and instead of 3 I was using sudo -i instead to become root, i will try again in the next week or tweo., thanks for writing it up! my actual goal at this point is to see if i can create a sysext package to use without installing dev tools....

I also found when using sudo -i one had to deconfigure the app service fully to be able to run install-dev-tools