ai2ys

Useful (Python) Libraries

Library	Description	Links
JAX	High-performance array computing

ai2ys

Retrieving disk usage information

Retrieving the disk space/usage

df -H

Retrieving file and folder sizes

Sorted by file/folder size

ai2ys

TensorFlow - Best Practices

Randomization

Understanding Randomization in TF Datasets (including TFRecords) https://colab.research.google.com/github/christianmerkwirth/colabs/blob/master/Understanding_Randomization_in_TF_Datasets.ipynb

Transfer learning/fine-tuning

How to apply transfer learning and fine-tuning. https://www.tensorflow.org/guide/keras/transfer_learning

ai2ys

Docker Context

Create a Docker context for remote Machine (access via SSH).

Create Docker context

# specifiy remote user, remote IP address, context name, and the description
remote_user=<user name>
remote_ip_address=<ip address of remote system>
context_name=<name for the context>

ai2ys

Install docker-compose on Ubuntu 20.04

I tried different installation methods via

1. curl which failed because of a non existing URL
2. apt-get which worked but the installed version does not GPU access
3. pip which worked and the installed version does GPU access

Below the tried methods in detail.

❌ Installing via curl - failed because of URL (404)

Following the instructions from https://docs.docker.com/compose/install/ resulted in the following error when trying to run docker-compose afterwards.

ai2ys

Pre-Built TensorFlow Docker Images Supporting CUDA 11

Newer NVIDA GPUs like the GeForce RTX 3090 require CUDA 11.

Pre-build Docker images provided by TensorFlow

TensorFlow provides pre-built Docker images on dockerhub - tensorflow/tensorflow. The underlying Docker files are availble in the TensorFlow GitHub repository.

Currently latest TF version is 2.7. In table gives an overview of CUDA version used within the pre-built Docker images for the specified TF versions. Versions were retrieved from the TensorFlow GitHub repository (date 2022-01-10).

ai2ys

Specify Package Version apt-get install

Installing a specific version

Install exactly the specified version, example:

# Installing exactly version '1.20.3-1ubuntu2' of wget
sudo apt-get install wget=1.20.3-1ubuntu2

ai2ys

ai2ys

Moving the Docker data directory - Ubuntu 20.04

Resources

• https://www.guguweb.com/2019/02/07/how-to-move-docker-data-directory-to-another-location-on-ubuntu/
• https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file

# work as root
sudo su

ai2ys

Using nvidia-smi

Commands on local system

Running nvidia-smi every n seconds

nvidia-smi -l <n seconds>

Retrieving GPU index and name