Setting up podman + nvidia on F37 - F40 (Self notes)

main.md

Follow the following instructions:

1. Install the C compiler through sudo dnf group install "C Development Tools and Libraries"
2. Install the kernel headers sudo dnf install kernel-devel
3. Download the latest drivers (replace with URL from nvidia website)

   wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.54.14/NVIDIA-Linux-x86_64-550.54.14.run
   chmod a+x NVIDIA-Linux-x86_64-550.54.14.run
   sudo ./NVIDIA-Linux-x86_64-550.54.14.run
   

4. Installer creates a disable config for nouveau.
5. Run sudo dracut --force to rebuild the initramfs to exclude nouveau
6. Actually run the install this time: sudo ./NVIDIA-Linux-x86_64-550.54.14.run
7. Confirm that you can nvidia-smi now
8. Add the container toolkit repos

   sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/fedora37/x86_64/cuda-fedora37.repo
   sudo dnf clean expire-cache
   sudo dnf install -y nvidia-container-toolkit-base
   

   Alternative:

   curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
   sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
   sudo dnf install -y nvidia-container-toolkit
   

9. Make sure the toolkit is installed nvidia-ctk --version
10. Generate the CDI: sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
11. Make sure the CDI is world readable: sudo chmod a+r /etc/cdi/nvidia.yaml (If not done you get this error Error: setting up CDI devices: unresolvable CDI devices nvidia.com/gpu=all later. Related to this)
12. Make sure devices are discovered via grep " name:" /etc/cdi/nvidia.yaml. Output should be a list of GPU IDs and name: all
13. Install podman sudo dnf install podman
14. Let the containers access the devices sudo chcon -t container_file_t /dev/nvidia* by fixing SELinux Context (If not done you get this error Failed to initialize NVML: Insufficient Permissions later. See here) NOTE: This needs to be rerun on every reboot.
15. Run this to check podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi -L. You should get: GPU 0: NVIDIA (...) (UUID: ...)

That was so helpful! Thank you a thousand times.

Nice one.

Next step is look for images in cuda/doc/supported-tags.md

podman pull docker.io/nvidia/cuda:12.2.0-devel-ubuntu22.04    # download devel 12.2 image
podman run --rm --device=nvidia.com/gpu=all -it docker.io/nvidia/cuda:12.2.0-devel-ubuntu22.04 nvidia-smi -L  # check container sees GPU, output should be "GPU 0: NVIDIA ..."
podman run --rm --device=nvidia.com/gpu=all -it docker.io/nvidia/cuda:12.2.0-devel-ubuntu22.04 bash

FWIW if after an upgrade the container stops seeing the driver:

#  ls -la /dev/nvidia*
ls: cannot access '/dev/nvidia-modeset': Permission denied
ls: cannot access '/dev/nvidia-uvm': Permission denied
ls: cannot access '/dev/nvidia-uvm-tools': Permission denied
ls: cannot access '/dev/nvidia0': Permission denied
ls: cannot access '/dev/nvidiactl': Permission denied

Add --security-opt label=disable to podman run or podman create. But mind the security hole if you are running 3rd party stuff.

It's working nicely, thank you!

Thanks, got it up and running!

Thanks!

This still works great, after installing the nvidia drivers through RPM Fusion on Fedora: https://rpmfusion.org/Howto/NVIDIA