Proper CUDA and cuDNN installation

cuda-install.md

date	title	tags
2020-02-29	Proper CUDA and cuDNN installation	tech

You're here, so you're probably already hurting because of CUDA and cuDNN compatibility, and I won't have to motivate you, or explain why you'd want to have standalone CUDA and cuDNN installations, if you're going to develop using Tensorflow in the long term.

1. Download your CUDA runfile (local) packages

Check out the TF compatibility matrix. Then hit NVidia's CUDA toolkit archive. Get your local runfiles.

wget https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda_11.0.3_450.51.06_linux.run
wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run
wget https://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run
wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run
wget https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda_11.5.1_495.29.05_linux.run
wget https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run

2. Install the toolkits only

sudo bash cuda_10.1.243_418.87.00_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-10.1.243 --librarypath=/usr/local/cuda-10.1.243

sudo bash cuda_10.2.89_440.33.01_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-10.2.89 --librarypath=/usr/local/cuda-10.2.89

sudo bash cuda_11.0.3_450.51.06_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-11.0.3 --librarypath=/usr/local/cuda-11.0.3

sudo bash cuda_11.2.2_460.32.03_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-11.2.2 --librarypath=/usr/local/cuda-11.2.2

sudo bash cuda_11.5.1_495.29.05_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-11.5.1 --librarypath=/usr/local/cuda-11.5.1

sudo bash cuda_11.6.0_510.39.01_linux.run --no-man-page --override --silent \
  --toolkit --toolkitpath=/usr/local/cuda-11.6.0 --librarypath=/usr/local/cuda-11.6.0

3. Download and extract your cuDNN tarballs

wget https://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.1-linux-x64-v7.6.5.32.tgz
wget https://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.2-linux-x64-v7.6.5.32.tgz
wget https://developer.download.nvidia.com/compute/redist/cudnn/v8.0.5/cudnn-11.0-linux-x64-v8.0.5.39.tgz
wget https://developer.download.nvidia.com/compute/redist/cudnn/v8.1.1/cudnn-11.2-linux-x64-v8.1.1.33.tgz
wget https://developer.download.nvidia.com/compute/redist/cudnn/v8.3.2/local_installers/11.5/cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz

sudo mkdir /usr/local/cudnn-10.1-7.6.5.32
sudo tar -xzf cudnn-10.1-linux-x64-v7.6.5.32.tgz -C /usr/local/cudnn-10.1-7.6.5.32 --strip 1

sudo mkdir /usr/local/cudnn-10.2-7.6.5.32
sudo tar -xzf cudnn-10.2-linux-x64-v7.6.5.32.tgz -C /usr/local/cudnn-10.2-7.6.5.32 --strip 1

sudo mkdir /usr/local/cudnn-11.0-8.0.5.39
sudo tar -xzf cudnn-11.0-linux-x64-v8.0.5.39.tgz -C /usr/local/cudnn-11.0-8.0.5.39 --strip 1

sudo mkdir /usr/local/cudnn-11.2-8.1.1.33
sudo tar -xzf cudnn-11.2-linux-x64-v8.1.1.33.tgz -C /usr/local/cudnn-11.2-8.1.1.33 --strip 1

4. Run your application or training with LD_LIBRARY_PATH

Your TensorFlow-using application will load the TF language support .so, which will load libtensorflow.so, which will dynamically load the various libraries.

We'll use the LD_LIBRARY_PATH environmental variable to tell the dynamic library loader where to look for the shared library files, first. That way we'll force Tensorflow to load the very specific CUDA and cuDNN libraries that are compatible with it.

In addition, the Tensorflow developers seem to like hardcoding paths and values, such as the path for the ptxas binary, and thus you'll encounter this error.

2020-03-01 13:19:42.121134: W tensorflow/stream_executor/cuda/redzone_allocator.cc:312] Not found: ./bin/ptxas not found
Relying on driver to perform ptx compilation. This message will be only logged once.

Then your program will hang for a good minute or so. So nice.

As @DawyD informs us in his comment to #33375, the currently accepted solution to this issue is to symbolically link to the CUDA version's bin directory from the current working directory from which you execute your Tensorflow application. Clarification.

You remember the TF compatibility matrix?

If you hit

2020-03-04 18:49:06.955169: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

you need to work around that Tensorflow issue by setting this environmental variable:

export TF_FORCE_GPU_ALLOW_GROWTH=true

or alternatively you can modify your Python code with

physical_devices = tf.config.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

TensorFlow 1.15.0

$ ln -snf /usr/local/cuda-10.0.130/bin bin
$ LD_LIBRARY_PATH=/usr/local/cuda-10.0.130/lib64:/usr/local/cudnn-10.0-7.4.2.24/lib64 python -c 'import tensorflow'

TensorFlow 2.0.0

$ ln -snf /usr/local/cuda-10.1.243/bin bin
$ LD_LIBRARY_PATH=/usr/local/cuda-10.1.243/lib64:/usr/local/cuda-10.1.243/extras/CUPTI/lib64:/usr/local/cudnn-10.1-7.6.5.32/lib64 python -c 'import tensorflow'

TensorFlow 2.1.0

$ ln -snf /usr/local/cuda-10.2.89/bin bin
$ LD_LIBRARY_PATH=/usr/local/cuda-10.2.89/lib64:/usr/local/cuda-10.2.89/extras/CUPTI/lib64:/usr/local/cudnn-10.2-7.6.5.32/lib64 python -c 'import tensorflow'

TensorFlow 2.4.1

$ ln -snf /usr/local/cuda-11.0.3/bin bin
$ LD_LIBRARY_PATH=/usr/local/cuda-11.0.3/lib64:/usr/local/cuda-11.0.3/extras/CUPTI/lib64:/usr/local/cudnn-11.0-8.0.5.39/lib64 python -c 'import tensorflow'

TensorFlow 2.5 nightly

$ ln -snf /usr/local/cuda-11.2.2/bin bin
$ LD_LIBRARY_PATH=/usr/local/cuda-11.2.2/lib64:/usr/local/cuda-11.2.2/extras/CUPTI/lib64:/usr/local/cudnn-11.2-8.1.1.33/lib64 python -c 'import tensorflow'

Your directory for the tarball for cuda 11 after the -C arg is not correctly set

Your directory for the tarball for cuda 11 after the -C arg is not correctly set

Thanks, now it is.

After a hole day of trouble I was finally able to get this running. I don't know what Nvidia messed up for the 20.04 installations but it is oddly frustrating. Thanks a lot for this gist.

Reworked and updated links as for 2022 - gist is here
This stores files in a different path (provided in STORAGE_PATH and creates symlink in LIB_PATH.

#!/bin/bash
# inspired by and used https://gist.github.com/mikaelhg/cae5b7938aa3dfdf3d06a40739f2f3f4 as a base

LIB_PATH=/usr/local
STORAGE_PATH=/path/to/big/drive/usr_local

BASE_DOWNLOAD_HREF=https://developer.download.nvidia.com/compute

# Exit on first error (not always, but most of cases)
set -euxo pipefail

# Prerequisites
sudo apt install zlib1g
sudo apt install xz-utils

symlink_to_more_major_versions () {
    cuda_folder_path=$1
    curr_cuda_folder_path=$cuda_folder_path
    while true; do
        more_major_version_cuda_folder_path=$(echo $curr_cuda_folder_path | rev | cut -d'.' -f2- | rev)
        sudo rm -rf $more_major_version_cuda_folder_path
        sudo ln -s $curr_cuda_folder_path $more_major_version_cuda_folder_path
        curr_cuda_folder_path=$more_major_version_cuda_folder_path
        [ -z "$(echo $(basename $curr_cuda_folder_path) | grep '\.')" ] && {
            break
        }
    done
}
download_extract_move_and_symlink_cuda() {
    cuda_href=$1
    echo "Downloading $cuda_href ..."
    cuda_fname=$(basename $cuda_href)
    cuda_folder=$(echo $cuda_fname | cut -d'_' -f-2 | sed 's/_/-/g')

    wget -nc $cuda_href
    echo "Trying to run $cuda_fname ..."
    echo "Executing command sudo bash $cuda_fname --no-man-page --override --silent --toolkit --toolkitpath=$LIB_PATH/$cuda_folder --librarypath=$LIB_PATH/$cuda_folder"
    sudo bash $cuda_fname --no-man-page --override --silent --toolkit --toolkitpath="$LIB_PATH/$cuda_folder" --librarypath="$LIB_PATH/$cuda_folder"
    echo "Now moving to $STORAGE_PATH in order to leave more space..."
    sudo cp -r "$LIB_PATH/$cuda_folder" "$STORAGE_PATH/"
    echo "Now REMOVING original location $LIB_PATH/$cuda_folder !!!"
    sudo rm -rf $LIB_PATH/$cuda_folder
    echo "Creating symlink to moved folder back in $LIB_PATH ..."
    sudo ln -s "$STORAGE_PATH/$cuda_folder" "$LIB_PATH/$cuda_folder"
    echo "Symlink created!"
    symlink_to_more_major_versions "$LIB_PATH/$cuda_folder"
}

# v11.7 is first CUDA version featuring 22.04 installer

for cuda_href in \
https://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run \
https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.0.3/local_installers/cuda_11.0.3_450.51.06_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.5.1/local_installers/cuda_11.5.1_495.29.05_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.5.2/local_installers/cuda_11.5.2_495.29.05_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.7.1/local_installers/cuda_11.7.1_515.65.01_linux.run \
https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run \
https://developer.download.nvidia.com/compute/cuda/12.0.0/local_installers/cuda_12.0.0_525.60.13_linux.run \
; do
    download_extract_move_and_symlink_cuda $cuda_href
done

#extract_cudnn_nvidia_way() {
#    cudnn_fname=$1
#    [ -n "$(echo $cudnn_fname | grep 'tar\.gz')" ] && {
#        cudnn_folder=${cudnn_fname%.tar.gz}
#        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xzf $cudnn_fname __"
#        mkdir -p $cudnn_folder
#        tar -xzf $cudnn_fname -C $cudnn_folder
#    }
#    [ -n "$(echo $cudnn_fname | grep '\.tgz')" ] && {
#        cudnn_folder=${cudnn_fname%.tgz}
#        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xzf $cudnn_fname __"
#        mkdir -p $cudnn_folder
#        tar -xzf $cudnn_fname -C $cudnn_folder
#    }
#    [ -n "$(echo $cudnn_fname | grep 'tar\.xz')" ] && {
#        cudnn_folder=${cudnn_fname%.tar.xz}
#        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xf $cudnn_fname __"
#        mkdir -p $cudnn_folder
#        tar -xf $cudnn_fname -C $cudnn_folder
#    }
#    [ -n "$(echo $cudnn_folder | grep archive)" ] && {
#        cudnn_info_str=$(echo $cudnn_folder | cut -d'-' -f4)
#        cudnn_version=$(echo $cudnn_info_str | cut -d'_' -f1)
#        cuda_version=$(echo $cudnn_info_str | cut -d'a' -f2)
#    } || {
#        cuda_version=$(echo $cudnn_folder | cut -d'-' -f2)
#        cudnn_version=$(echo $cudnn_folder | cut -d'v' -f2)
#    }
#    echo "Now copying files from $cudnn_folder folder: CUDA version is $cuda_version, CuDNN version is $cudnn_version"
#    LIB_CUDA_FOLDER="$LIB_PATH/cuda-$cuda_version"
#    [ -d $LIB_CUDA_FOLDER ] || {
#        echo "$LIB_CUDA_FOLDER does not exist! Exiting."
#        exit 1
#    }
#    [ -d "$cudnn_folder/include" ] || {
#        [ -d "$cudnn_folder/cuda" ] && {
#            mv $cudnn_folder/cuda/* $cudnn_folder/
#        } || {
#            echo "Expecting $cudnn_folder/cuda folder: not found: exiting."
#            exit 1
#        }
#    }
#    sudo cp $cudnn_folder/include/cudnn*.h "$LIB_CUDA_FOLDER/include"
#    [ -d $cudnn_folder/lib ] && {
#        sudo cp -P $cudnn_folder/lib/libcudnn* "$LIB_CUDA_FOLDER/lib64"
#    }
#    [ -d $cudnn_folder/lib64 ] && {
#        sudo cp -P $cudnn_folder/lib64/libcudnn* "$LIB_CUDA_FOLDER/lib64"
#    }
#    sudo chmod a+r $LIB_CUDA_FOLDER/include/cudnn*.h $LIB_CUDA_FOLDER/lib64/libcudnn*
#}

extract_cudnn_into_its_own_folder() {
    cudnn_href=$1
    [ -n "$(echo $cudnn_href | grep 'local_installers')" ] && {
        cuda_version=$(echo $cudnn_href | rev | cut -d'/' -f2 | rev)
    } || {
        cuda_version=''
    }
    cudnn_fname=$(basename $cudnn_href)
    [ -n "$(echo $cudnn_fname | grep 'tar\.gz')" ] && {
        cudnn_folder=${cudnn_fname%.tar.gz}
        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xzf $cudnn_fname __"
        mkdir -p $cudnn_folder
        tar -xzf $cudnn_fname -C $cudnn_folder
    }
    [ -n "$(echo $cudnn_fname | grep '\.tgz')" ] && {
        cudnn_folder=${cudnn_fname%.tgz}
        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xzf $cudnn_fname __"
        mkdir -p $cudnn_folder
        tar -xzf $cudnn_fname -C $cudnn_folder
    }
    [ -n "$(echo $cudnn_fname | grep 'tar\.xz')" ] && {
        cudnn_folder=${cudnn_fname%.tar.xz}
        echo "Unpacking $cudnn_fname into $cudnn_folder using command __ tar -xf $cudnn_fname __"
        mkdir -p $cudnn_folder
        tar -xf $cudnn_fname -C $cudnn_folder
    }
    [ -n "$(echo $cudnn_folder | grep archive)" ] && {
        cudnn_info_str=$(echo $cudnn_folder | cut -d'-' -f4)
        cudnn_version=$(echo $cudnn_info_str | cut -d'_' -f1)
        [ -z "$cuda_version" ] && {
            cuda_version=$(echo $cudnn_info_str | cut -d'a' -f2)
            echo "Cuda version is $cuda_version."
        } || {
            echo "Cuda version is $cuda_version."
        }
    } || {
        [ -z "$cuda_version" ] && {
            cuda_version=$(echo $cudnn_folder | cut -d'-' -f2)
            echo "Cuda version is $cuda_version."
        } || {
            echo "Cuda version is $cuda_version."
        }
        cudnn_version=$(echo $cudnn_folder | cut -d'v' -f2)
    }

    [ -d "$cudnn_folder/include" ] || {
        [ -d "$cudnn_folder/cuda" ] && {
            mv $cudnn_folder/cuda/* $cudnn_folder/
        } || {
            echo "Expecting $cudnn_folder/cuda folder: not found: exiting."
            exit 1
        }
    }
    echo "$cudnn_fname : CUDA version is $cuda_version, CuDNN version is $cudnn_version"

    lib_cudnn_folder="cudnn-$cuda_version-$cudnn_version"
    sudo mkdir -p "$STORAGE_PATH/$lib_cudnn_folder"
    sudo cp -r $cudnn_folder/* "$STORAGE_PATH/$lib_cudnn_folder/"
    sudo rm -f "$LIB_PATH/$lib_cudnn_folder"
    sudo ln -s "$STORAGE_PATH/$lib_cudnn_folder" "$LIB_PATH/$lib_cudnn_folder"
    sudo chmod a+r $STORAGE_PATH/$lib_cudnn_folder/include/cudnn*.h
    echo "Now testing existence of $STORAGE_PATH/$lib_cudnn_folder/lib64 existence."
    [ -d "$STORAGE_PATH/$lib_cudnn_folder/lib64" ] && {
        sudo chmod a+r $STORAGE_PATH/$lib_cudnn_folder/lib64/libcudnn*
    } || {
        [ -d "$STORAGE_PATH/$lib_cudnn_folder/lib" ] && {
            sudo chmod a+r $STORAGE_PATH/$lib_cudnn_folder/lib/libcudnn*
        } || {
            echo "lib or lib64 not found: exiting."
            exit 1
        }
    }
}

download_and_extract_cudnn() {
    cudnn_href=$1
    wget -nc $cudnn_href
    extract_cudnn_into_its_own_folder $cudnn_href
    #cudnn_fname=$(basename $cudnn_href)
    #extract_cudnn_nvidia_way $cudnn_fname
}

OFS=$IFS
IFS=$' '
for cudnn_href in \
https://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.1-linux-x64-v7.6.5.32.tgz \
https://developer.download.nvidia.com/compute/redist/cudnn/v7.6.5/cudnn-10.2-linux-x64-v7.6.5.32.tgz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.4.1/local_installers/10.2/cudnn-linux-x86_64-8.4.1.50_cuda10.2-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.5.0/local_installers/10.2/cudnn-linux-x86_64-8.5.0.96_cuda10-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.0.5/cudnn-11.0-linux-x64-v8.0.5.39.tgz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.1.1/cudnn-11.2-linux-x64-v8.1.1.33.tgz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.3.2/local_installers/11.5/cudnn-linux-x86_64-8.3.2.44_cuda11.5-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.3.3/local_installers/11.5/cudnn-linux-x86_64-8.3.3.40_cuda11.5-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.4.1/local_installers/11.6/cudnn-linux-x86_64-8.4.1.50_cuda11.6-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.5.0/local_installers/11.7/cudnn-linux-x86_64-8.5.0.96_cuda11-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.6.0/local_installers/11.8/cudnn-linux-x86_64-8.6.0.163_cuda11-archive.tar.xz \
https://developer.download.nvidia.com/compute/redist/cudnn/v8.7.0/local_installers/11.8/cudnn-linux-x86_64-8.7.0.84_cuda11-archive.tar.xz ; do
    echo "Link $cudnn_href"
    download_and_extract_cudnn $cudnn_href
done
IFS=$OFS

okay