Mehdi Cherti mehdidc

Ubuntu 22.04 for Deep Learning

In the name of God

This gist contains steps to setup Ubuntu 22.04 for deep learning.

Install Ubuntu 22.04

Specific Papers

https://arxiv.org/abs/1505.07376

Texture Synthesis Using Convolutional Neural Networks
https://arxiv.org/abs/1508.06576

A Neural Algorithm of Artistic Style
https://arxiv.org/abs/1603.03417

Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
https://arxiv.org/abs/1701.02096

Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Texture Synthesis

(Internal Tranining Material)

Usually the first step in performance optimization is to do profiling, e.g. to identify performance hotspots of a workload. This gist tells basic knowledge of performance profiling on PyTorch, you will get:

How to find the bottleneck operator?
How to trace source file of a particular operator?
How do I indentify threading issues? (oversubscription)
How do I tell a specific operator is running efficiently or not?

This tutorial takes one of my recent projects - pssp-transformer as an example to guide you through path of PyTorch CPU peformance optimization. Focus will be on Part 1 & Part 2.

Multi-node-training on slurm with PyTorch

What's this?

A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
Warning: might need to re-factor your own code.
Warning: might be secretly condemned by your colleagues because using too many GPUs.

Document Revision 1.2 [07/15/22]

Author Notes: Thanks to everyone who has been testing or using this installation process. I've refined the instructions to try to make this installation as simple and cross platform as humanly possible. I've also updated the Steam Deck instructions now that I have had mine for a while and thoroughly tested the installation process.

Basic Game Installation

Final Fantasy IX should be installed like any standard steam game through the steam client. Ragarding compatibility layers: it should work with a stable release of Proton 7 (7.0.X recommended) through the steam client. This also includes controller support if you are using Steam's native controller configurations. If you have any problems with a given Proton release, I would also recommend trying the latest GloriousEggroll proton builds to se

Hello,

When you look at a repo, be sure to use git log --all and look at the latest commits across all branches. You would see that my work is on the "dec6" branch, not the main branch.

In general, it's more reliable to DM me on twitter than to email me. I'll usually respond within a day. Emails are hit-or-miss.

Here is the critical difference:

	from __future__ import print_function

	import argparse
	import torch.backends.cudnn as cudnn
	import torch.nn.functional as F
	import torch.optim as optim
	import torch.utils.data.distributed
	from torchvision import models
	import horovod.torch as hvd
	import timeit

	# mainly from:
	# 1. https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars
	# 2. https://github.com/NVIDIA/DeepLearningExamples/issues/57
	# 3. https://docs.nvidia.com/deeplearning/frameworks/tensorflow-user-guide/index.html#variablesaddtf

	def is_using_hvd():
	env_vars = ["OMPI_COMM_WORLD_RANK", "OMPI_COMM_WORLD_SIZE"]

	if all([var in os.environ for var in env_vars]):
	return True

	set nocompatible " be iMproved, required
	let g:python3_host_prog = '/usr/local/opt/[email protected]/bin/python3.8'

	if empty(glob('~/.vim/autoload/plug.vim'))
	silent !curl -fLo ~/.vim/autoload/plug.vim --create-dirs
	\ https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
	autocmd VimEnter * PlugInstall --sync \| source $MYVIMRC
	endif

	call plug#begin('~/.vim/plugged')

	#!/usr/bin/env bash
	###
	# NB: You probably don't want this gist any more.
	# Instead, use this version from `fastsetup`:
	# https://github.com/fastai/fastsetup/blob/master/setup-conda.sh
	###

	set -e

	cd