All the stuff to get CUDA 10.1 working with NVIDIA GPUs on Ubuntu 18.04. My notes.
sudo apt install nvidia-driver-430- reboot
- run
nvidia-smi. If it does not show your GPU, stop, fix. If this doesn't work, nothing else will (the rest of the stuff will compile, but won't work)
- download the right CUDA local deb file from https://developer.nvidia.com/cuda-downloads
For me it was:
wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67_1.0-1_amd64.deb
- Now install it
sudo dpkg -i cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67_1.0-1_amd64.deb # change package name if yours is different
sudo apt-key add /var/cuda-repo-10-1-local-10.1.168-418.67/7fa2af80.pub # change package name if yours is different
sudo apt-get update
sudo apt install cuda-toolkit-10-1
- Make sure CUDA paths are set correctly for all users on log-in
Create a new file, /etc/profile.d/cuda.sh
Add the following (make sure paths are correct in your system)
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/lib:$LD_LIBRARY_PATH
export CUDADIR=/usr/local/cuda
export CUDA_HOME=/usr/local/cuda
- Update cuda config
sudo mv /etc/ld.so.conf.d/cuda-10-1.conf /etc/ld.so.conf.d/cuda-10-1.conf-orig
Create new cuda.conf:
echo "/usr/local/cuda/lib64" > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig
Get the paths imported in your curent system:
source /etc/profile.d/cuda.sh
Install deps (you may not need all of these if you don't plan to install all the ML stuff I needed later)
sudo apt-get install build-essential cmake libjpeg-dev libpng-dev libtiff-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libgtk-3-dev libatlas-base-dev gfortran python3-dev pkg-config unzip ffmpeg qtbase5-dev python3-dev python3-numpy libhdf5-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libtiff5-dev libtesseract-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer-plugins-base1.0-0 libgstreamer-plugins-base1.0-dev libpng16-16 libpng-dev libv4l-dev libtbb-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev v4l-utils libleptonica-dev
download & install cuDNN development & runtime library from NVIDIA runtime. For my system it was :
Runtime:
https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7_7.6.5.32-1%2Bcuda10.1_amd64.deb
Development:
https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.1_20191031/Ubuntu18_04-x64/libcudnn7-dev_7.6.5.32-1%2Bcuda10.1_amd64.deb
After downloading, install them (you can also use dpkg -i)
gdebi libcudnn7_7.6.5.32-1+cuda10.1_amd64.deb gdebi libcudnn7-dev_7.6.5.32-1+cuda10.1_amd64.deb
Compiling OpenCV 4.2.x with GPU support (if you need it)
--------------------------------------------------------
(Let's assume my cwd is /home/pp/fiddle/opencv)
git clone https://github.com/opencv/opencv git clone https://github.com/opencv/opencv_contrib
cd opencv mkdir build cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE
-D INSTALL_PYTHON_EXAMPLES=ON
-D INSTALL_C_EXAMPLES=OFF
-D OPENCV_ENABLE_NONFREE=ON
-D WITH_CUDA=ON
-D WITH_CUDNN=ON
-D OPENCV_DNN_CUDA=ON
-D ENABLE_FAST_MATH=1
-D CUDA_FAST_MATH=1
-D WITH_CUBLAS=1
-D OPENCV_EXTRA_MODULES_PATH=/home/pp/fiddle/opencv//opencv_contrib/modules
-D HAVE_opencv_python3=ON
-D PYTHON_EXECUTABLE=/usr/bin/python3
-D BUILD_EXAMPLES=ON ..
make -j$(nproc)
make install
Note that according to Adrian's pyimagesearch [article](https://www.pyimagesearch.com/2020/02/03/how-to-use-opencvs-dnn-module-with-nvidia-gpus-cuda-and-cudnn/), you also need to add `-D CUDA_ARCH_BIN=XX` where `XX` is your GPU architecture version. However, in my tests, this is not needed and is automatically detected.
If you plan to compile FFMPEG from source to support CUDA you need to copy the package config
sudo cp unix-install/opencv4.pc /usr/lib/pkgconfig/
Compiling Tensorflow with GPU support (if you need it)
----------------------------------------------------------
sudo apt-get install libcupti-dev add to LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH wget https://github.com/bazelbuild/bazel/releases/download/0.26.1/bazel-0.26.1-installer-linux-x86_64.sh
install bazel via the script above
install tensorflow RT (needs free NVIDIA login)
Compiling Darknet with GPU support (if you need it)
----------------------------------------------------------
https://github.com/AlexeyAB/darknet (take Alexy's fork, pjreddie main repo has OCV4 compile issues, PR open)
Make file mods:
GPU=1 OPENCV=1 LIBSO=1
change `pkg-config to pkg-config --libs opencv4`
YoloV3 modify cfg for 4GB, otherwise trying to load YoloV3 into a 4GB card will seg fault. With these changes,
YoloV3 model takes 1.6GB memory
[net] batch=1 subdivisions=1 width=416 height=416
Compiling Dlib with GPU support (if you need it)
------------------------------------------------------
Easiest of the lot. After you have everything running, GPU working fine, etc. just do
git clone https://github.com/davisking/dlib cd dlib python ./setup.py install
That's all. It will auto configure everything. Make sure its confiured with GPU
In a python shell:
import dlib dlib.DLIB_USE_CUDA True print (dlib.cuda.get_num_devices()) 1
Installing face-recognition with GPU support (if you need it)
-----------------------------------------------------------------
Just do a `pip install face-recognition` after you have installed a GPU version of dlib.
FFMPEG with CUDA (if you need it)
----------------------------------
Follow instructions at https://developer.nvidia.com/ffmpeg
If you need hardware encode support, download and install https://developer.nvidia.com/nvidia-video-codec-sdk first
Here were all my ffmpeg build flags: (note that the link above has lesser flags, I needed more libs for other software)
./configure --enable-cuda --enable-cuvid --enable-nvenc --enable-nonfree --enable-libnpp --enable-vaapi --extra-cflags=-I/usr/local/cuda/include --extra-ldflags=-L/usr/local/cuda/lib64 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-libass --enable-libcaca --enable-libcdio --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libx264 --enable-shared --enable-nvdec
Then do:
make sudo make install
FFMPEG with CUDA with ZoneMinder (if you need it)
---------------------------------------------------
Finally, I use the following macro to build ZM from source, so it builds/links with the right libs:
alias zm_cmake='git submodule update --init --recursive && cmake -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_SKIP_RPATH=ON -DCMAKE_VERBOSE_MAKEFILE=OFF -DCMAKE_COLOR_MAKEFILE=ON -DZM_RUNDIR=/var/run/zm -DZM_SOCKDIR=/var/run/zm -DZM_TMPDIR=/var/tmp/zm -DZM_LOGDIR=/var/log/zm -DZM_WEBDIR=/usr/share/zoneminder/www -DZM_CONTENTDIR=/var/cache/zoneminder -DZM_CGIDIR=/usr/lib/zoneminder/cgi-bin -DZM_CACHEDIR=/var/cache/zoneminder/cache -DZM_WEB_USER=www-data -DZM_WEB_GROUP=www-data -DCMAKE_INSTALL_SYSCONFDIR=etc/zm -DZM_CONFIG_DIR=/etc/zm -DCMAKE_BUILD_TYPE=Release'
So I do:
sudo service zoneminder stop zm_cmake (in the source root of ZM) make sudo make install