I needed to do:
sudo rmmod nvidia_uvm
sudo modprobe nvidia_uvm
From the nvidia forums.
I also got the following error fixed (docker-nvidia related) with these commands:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: file creation failed: /var/lib/docker/overlay2/a5a8483bf9826a94713d11ea85bee8ed790996218e612b10de9dd2d80f39d7af/merged/dev/nvidia-uvm: input/output error: unknown.
thx bro