Ubuntu 20.04 for Deep Learning.md

Ubuntu 20.04 for Deep Learning

In the name of God

This gist contains steps to setup Ubuntu 20.04 for deep learning.

---

Install Ubuntu 20.04:

• Computer name: Name-PC
• Name: Name
• User name: name
• Password: ********

---

Update Ubuntu:

$ sudo apt update
$ sudo apt full-upgrade --yes
$ sudo apt autoremove --yes
$ sudo apt autoclean --yes
$ reboot

---

Create Update Script:

• Create a file (~/full-update.sh) with the following lines:

#!/bin/bash

if [ "$EUID" -ne 0 ]
  then echo "Error: Please run as root."
  exit
fi

clear

echo "################################################################################"
echo "Updating list of available packages..."
echo "--------------------------------------------------------------------------------"
apt update
echo "################################################################################"
echo

echo "################################################################################"
echo "Upgrading the system by removing/installing/upgrading packages..."
echo "--------------------------------------------------------------------------------"
apt full-upgrade --yes
echo "################################################################################"
echo

echo "################################################################################"
echo "Removing automatically all unused packages..."
echo "--------------------------------------------------------------------------------"
apt autoremove --yes
echo "################################################################################"
echo

echo "################################################################################"
echo "Clearing out the local repository of retrieved package files..."
echo "--------------------------------------------------------------------------------"
apt autoclean --yes
echo "################################################################################"
echo

---

Change Settings:

• Review Ubuntu Settings

---

Change Software & Updates:

• Review Ubuntu Software & Updates

---

Update Ubuntu:

• $ sudo ~/full-update.sh

---

Install Chrome (https://www.google.com/chrome):

• $ sudo dpkg -i google-chrome-stable_current_amd64.deb

---

Install Development Tools:

• $ sudo apt install build-essential pkg-config cmake cmake-qt-gui ninja-build valgrind

---

Install MinGW Tools:

$ sudo apt install mingw-w64 mingw-w64-tools
$ sudo apt install binutils-mingw-w64

$ sudo update-alternatives --set i686-w64-mingw32-gcc /usr/bin/i686-w64-mingw32-gcc-posix
$ sudo update-alternatives --set i686-w64-mingw32-g++ /usr/bin/i686-w64-mingw32-g++-posix
 
$ sudo update-alternatives --set x86_64-w64-mingw32-gcc /usr/bin/x86_64-w64-mingw32-gcc-posix
$ sudo update-alternatives --set x86_64-w64-mingw32-g++ /usr/bin/x86_64-w64-mingw32-g++-posix

---

Install ARM Tools:

$ sudo apt install crossbuild-essential-armel crossbuild-essential-armhf crossbuild-essential-arm64
$ sudo apt install binutils-multiarch binutils-arm-none-eabi binutils-arm-linux-gnueabi binutils-arm-linux-gnueabihf binutils-aarch64-linux-gnu

---

Install Git:

$ sudo apt install git
$ git config --global user.name "Name"
$ git config --global user.email "[email protected]"
$ git config --global core.editor "gedit -s"

• Copy your own SSH keys to ~/.ssh

---

Install Python 3:

• $ sudo apt install python3 python3-wheel python3-pip python3-venv python3-dev python3-setuptools

---

Install NVIDIA Drivers for Deep Learning:

• https://www.tensorflow.org/install/gpu
• TensorFlow and CUDA compatibilities:
  • https://www.tensorflow.org/install/source#gpu
• Check Hardware:
  • $ sudo lshw -C display
• Install NVIDIA GPU Driver:
  • Software & Updates > Additional Drivers > NVIDIA
  • Try $ sudo ubuntu-drivers autoinstall if NVIDIA drivers are disabled
• Install CUDA Toolkit (CUDA 11.2):
  • https://developer.nvidia.com/cuda-toolkit-archive
  1. Install prerequisites:
     • $ sudo apt install linux-headers-$(uname -r)
  2. Install cuda_11.2.2_460.32.03_linux.run (without Driver, use --override if gcc version not supported)
  3. Set up the development environment by modifying the PATH and LD_LIBRARY_PATH variables (~/.bashrc):
     • export PATH=$PATH:/usr/local/cuda-11.2/bin
     • export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64
  4. Check that GPUs are visible using the following command:
     • $ nvidia-smi
• Install cuDNN v8.1, for CUDA 11.2:
  • https://developer.nvidia.com/cudnn
  • https://developer.nvidia.com/rdp/cudnn-archive
  • $ sudo dpkg -i libcudnn8_8.1.1.33-1+cuda11.2_amd64.deb
  • $ sudo dpkg -i libcudnn8-dev_8.1.1.33-1+cuda11.2_amd64.deb
  • $ sudo dpkg -i libcudnn8-samples_8.1.1.33-1+cuda11.2_amd64.deb # Optional
• Reboot

---

Machine Learning Environment:

$ python3 -m venv ~/venv/ml
$ source ~/venv/ml/bin/activate
(ml) $ pip install --upgrade pip setuptools wheel
(ml) $ pip install --upgrade numpy scipy matplotlib ipython jupyter pandas sympy nose
(ml) $ pip install --upgrade scikit-learn scikit-image
(ml) $ deactivate

---

Deep Learning Environment (Tensorflow-CPU):

• https://www.tensorflow.org/install

$ python3 -m venv ~/venv/tfcpu
$ source ~/venv/tfcpu/bin/activate
(tfcpu) $ pip install --upgrade pip setuptools wheel
(tfcpu) $ pip install --upgrade opencv-python opencv-contrib-python
(tfcpu) $ pip install --upgrade tensorflow-cpu tensorboard keras
(tfcpu) $ deactivate

---

Deep Learning Environment (Tensorflow-GPU):

• https://www.tensorflow.org/install/gpu

$ python3 -m venv ~/venv/tfgpu
$ source ~/venv/tfgpu/bin/activate
(tfgpu) $ pip install --upgrade pip setuptools wheel
(tfgpu) $ pip install --upgrade opencv-python opencv-contrib-python
(tfgpu) $ pip install --upgrade tensorflow-gpu tensorboard keras
(tfgpu) $ deactivate

---

Check GPU Support by Tensorflow:

$ source ~/venv/tfgpu/bin/activate
(tfgpu) $ python
>>> from tensorflow.python.client import device_lib
>>> device_lib.list_local_devices()
>>> exit()
(tfgpu) $ deactivate

---

Additional Useful Python Packages:

• Project documentation with Markdown:
• https://www.mkdocs.org/
• https://squidfunk.github.io/mkdocs-material/

$ pip install mkdocs mkdocs-material

---

Install Miniconda:

Install:

$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ chmod +x Miniconda3-latest-Linux-x86_64.sh

$ ./Miniconda3-latest-Linux-x86_64.sh

$ # Do you wish the installer to initialize Miniconda3
  # by running conda init?
  # yes

$ source ~/miniconda3/bin/activate 
$ conda config --set auto_activate_base false
$ conda deactivate

Activate and Deactivate:

$ conda activate
(base) $ conda deactivate

Managing conda:

(base) $ conda info
(base) $ conda update conda

Managing environments:

(base) $ conda info --envs

(base) $ conda create --name snakes python=3.5
(base) $ conda info --envs

(base) $ conda activate snakes

(snakes) $ python --version

(snakes) $ conda search beautifulsoup4

(snakes) $ conda install beautifulsoup4
(snakes) $ conda list

(snakes) $ conda update beautifulsoup4

(snakes) $ conda uninstall beautifulsoup4
(snakes) $ conda list

(snakes) $ conda deactivate

(base) $ conda remove --name snakes --all

(base) $ conda info --envs

• https://conda.io/
• https://docs.conda.io/en/latest/miniconda.html
• https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html
• https://anaconda.org/search

---

Install Qt:

• https://www.qt.io/
• https://doc.qt.io/qt-6/linux.html
• $ sudo apt install build-essential libgl1-mesa-dev
• For Android Java 8 required.
  • https://doc.qt.io/qt-6/android-getting-started.html
  • $ sudo apt install openjdk-8-jre openjdk-8-jdk
  • $ sudo apt install libstdc++6 libncurses5
  • $ sudo apt install libsdl1.2debian
  • Install Android SDK & Android Studio

---

Install PyCharm:

• https://www.jetbrains.com/pycharm/

---

Enable GPU support for PyCharm Projects:

• Edit Configurations...
• Environment variables:
  • PATH=$PATH:/usr/local/cuda-11.2/bin
  • LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/extras/CUPTI/lib64

---

Install Docker Engine & Docker Compose:

• https://docs.docker.com/engine/install/ubuntu/
• https://docs.docker.com/compose/install/

Nvidia Container Toolkit:

• https://github.com/NVIDIA/libnvidia-container

• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/overview.html

• https://github.com/NVIDIA/nvidia-docker/

• https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

• https://hub.docker.com/r/nvidia/cuda

Make sure you have installed the NVIDIA driver and Docker engine for your Linux distribution Note that you do not need to install the CUDA Toolkit on the host system, but the NVIDIA driver needs to be installed

$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
   && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
   && curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
   
$ sudo apt update

$ sudo apt install --yes nvidia-docker2

$ sudo systemctl restart docker

# NOTE: --runtime=nvidia
$ docker run --rm --runtime=nvidia nvidia/cuda:11.0-base nvidia-smi
# OR
$ docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

TensorFlow Docker:

• https://www.tensorflow.org/install/docker
• https://hub.docker.com/u/tensorflow/tensorflow

# CPU
$ docker run -it tensorflow/tensorflow bash

# GPU
# NOTE: --runtime=nvidia
$ docker run --runtime=nvidia -it tensorflow/tensorflow:latest-gpu bash

Running ARM Docker Containers:

• https://en.wikipedia.org/wiki/QEMU
• https://en.wikipedia.org/wiki/Binfmt_misc
• https://wiki.debian.org/QemuUserEmulation
• https://github.com/multiarch/qemu-user-static
• https://github.com/docker-library/official-images#architectures-other-than-amd64

$ sudo apt install qemu qemu-user-static binfmt-support
$ # Host Architecture: x86_64
$ uname -m
$ # ARM Container Architecture: armv7l
$ docker run --rm arm32v7/debian uname -m

Model Converter:

• https://github.com/microsoft/MMdnn

$ docker run -it -v $(pwd):/models mmdnn/mmdnn:cpu.small

---

Install Additional Tools:

$ sudo apt install ubuntu-restricted-extras
$ sudo apt install virtualbox virtualbox-dkms virtualbox-ext-pack virtualbox-guest-additions-iso
$ sudo apt install curl wget uget unzip tar rar unrar
$ sudo apt install gimp
$ sudo apt install kdiff3

• Ubuntu Software > System Load Indicator

---

Install Additional Fonts:

• $ mkdir ~/.fonts
• Copy fonts to ~/.fonts

I am extremely new to Ubuntu I am trying to set up environment, so in the second step where you instructed to create '~/full-update.sh

' Can you help me. I am not understanding what has to be done.