MARCH Login.md

MARCH Course Computation Resources

Login

Command: ssh -J <username>@<login node> <username>@<GPU node>

Login nodes

• 140.112.51.229 (z-gotham.math.ntu.edu.tw)
• 140.112.51.228 (z-greenwich.math.ntu.edu.tw)

GPU nodes

• 140.112.51.222 (brillante.math.ntu.edu.tw)
• 140.112.51.223 (cantabile.math.ntu.edu.tw)

Example: ssh -J [email protected] [email protected]

The default password is the same as your username. Change the default password with yppasswd

Use Tensorflow

Load necessary modules:

• module load anaconda3/5.2.0
• module load cuda/10.0
• module load cudnn-7.6/10.0
• module load tensorflow-gpu/1.14.0

Run you Tensroflow Python code!

Use Chainer

Note: It only works on CPU. Load necessary modules:

• module load anaconda3/5.2.0
• module load cupy
• module load chainer

Select GPU

• Check which GPU is not in use with nvidia-smi (Remember the ID!)
• Run export CUDA_VISIBLE_DEVICES=<ID> to use the specific GPU

[General settings]
module load cuda/10.0 cudnn-7.6/10.0
export CUDA_VISIBLE_DEVICES=8
pip install sklearn

[TensorFlow]
pip install tensorflow-gpu

https://gist.github.com/guillaume-chevalier/6b01c4e43a123abf8db69fa97532993f
https://gist.github.com/giuseppebonaccorso/48694ea84a474ad8c748c240ed8d4376
https://medium.com/datadriveninvestor/part-7-review-of-gradients-hessians-and-newtons-method-with-examples-implemented-in-tensorflow-9a1798a4c33b
https://squadrick.github.io/journal/natural-gradients-in-tensorflow.html
pip install 'kfac[tensorflow_gpu]'
pip install sonnet graphs

[Horovod]
https://github.com/horovod/horovod

[PyTorch]
pip install https://download.pytorch.org/whl/cu100/torch-1.1.0-cp36-cp36m-linux_x86_64.whl
pip3 install https://download.pytorch.org/whl/cu100/torchvision-0.3.0-cp36-cp36m-linux_x86_64.whl

[natural-gradient]
Use python 3.6+
git clone https://github.com/wiseodd/natural-gradients
In input_data.py “urllib” > “urllib.request”
mkdir temp

You can also use the following commands to execute hands-on.

natural-gradient

module load anaconda3/5.2.0
module load torch

edit input_data.py
line 4 and line 15: urllib -> urllib.request

If you encountered permission denied error on pip install, try to add --user.

where to get the demonstrated .py files: full_fisher.py, kfac.py, adam.py ... etc ?

PyTorch > Chainer translation

torch.nn > chainer.links

Conv2D > Convolution2D
Linear > Linear

torch.nn.functional > chainer.functions

MaxPool2d > max_pooling_2d
relu > relu
CrossEntropyLoss > softmax_cross_entropy
backward > backward

y=model.forward(x) > y=model(x)

torch.optim > chainer.optimizers

SGD > SGD
RMSprop > RMSprop
Adagrad > AdaGrad
Adadelta > AdaDelta
Adam > Adam
step > update
zero_grad > cleargrads

Use ChainerKFAC

• git clone https://github.com/tyohei/chainerkfac
• cd chainerkfac
• pip install -e .
• cd examples/mnist
• pip install mpi4py

Use TensorFlowKFAC

• use python2.7
• module load cuda/9.0 cudnn-7.1/9.0
• pip install tensorflow-gpu==1.12.0
• pip install 'kfac[tensorflow-gpu]'
• pip install tensorflow-probability==0.5.0
• pip install dm-sonnet==1.32.0
• Add a line tf.logging.set_verbosity(tf.logging.INFO) to line 334 in autoencoder_mnist.py.

Use ChainerKFAC with MPI

module load anaconda3/5.2.0
module load cupy/cuda100
module load chainer-gpu
module load mpi4py/3.0.0-mpi-3.1.4

mpirun -np 2 python train.py --distributed
Note: for some reasons, you need to go to dolcissimo (140.112.51.224).

Use PyTorchKFAC

module load cuda/10.0 cudnn-7.6/10.0 torch
https://github.com/alecwangcq/KFAC-Pytorch.git
pip install networkx
pip install tensorflow-gpu
pip install torch torchvision

[email protected]