Mohammad Alyetama Alyetama

Terminal color definitions

FOREGROUND

BLACK   = \033[30m
RED     = \033[31m
GREEN   = \033[32m
YELLOW  = \033[33m
BLUE    = \033[34m
MAGENTA = \033[35m

Multi-node-training on slurm with PyTorch

What's this?

A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
Warning: might need to re-factor your own code.
Warning: might be secretly condemned by your colleagues because using too many GPUs.

Introduction

The purpose of this short howto is to show you how to:

use openconnect [1] to connect to an enterprise cisco anyconnect endpoint
whilst minimizing the amount of traffic that your route through the vpn connection

Usually VPN administrators will puth the default route to the users, so that all user traffic is routed through the vpn connection. This is to address the various security concerns around compromised user computers bridging external internet traffic into the secure VPN network.

While the VPN administrator can push routes to the clients, the client can ignore these default routes and establish client side routing so that only the required A.B.C.D/E network is routed through the VPN. All other traffic will still use the clients default route and default outbound internet connection.

A quick and dirty syntax translation / conversion reference guide to ease the transition between Python and Julia. This is not meant as a reference to the language. For that you should read the manual.

Some important differences

Arrays in Julia are indexed starting from 1.
In Julia classes (i.e. types) don't own methods. Methods are implementations of generic functions and are invoked in a "static style", i.e. instead of Python's str1.rstrip(), we will have rstrip( str1 ), instead of file1.close(), close( file1 ).

Some important similarities.

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h or --help depending on your psql version):

-E: will describe the underlaying queries of the \ commands (cool for learning!)
-l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

Virtual Hosts on nginx (CSC309)

When hosting our web applications, we often have one public IP address (i.e., an IP address visible to the outside world) using which we want to host multiple web apps. For example, one may wants to host three different web apps respectively for example1.com, example2.com, and example1.com/images on the same machine using a single IP address.

How can we do that? Well, the good news is Internet browsers

	# We'll be installing Homebrew in the /opt directory.
	cd /opt

	# Create a directory for Homebrew. This requires root permissions.
	sudo mkdir homebrew

	# Make us the owner of the directory so that we no longer require root permissions.
	sudo chown -R $(whoami) /opt/homebrew

	# Download and unzip Homebrew. This command can be found at https://docs.brew.sh/Installation.

	#!/usr/bin/env python3
	# python3 update of https://gist.github.com/dergachev/7028596
	# Create a basic certificate using openssl:
	# openssl req -new -x509 -keyout server.pem -out server.pem -days 365 -nodes
	# Or to set CN, SAN and/or create a cert signed by your own root CA: https://thegreycorner.com/pentesting_stuff/writeups/selfsignedcert.html

	import http.server
	import ssl

	httpd = http.server.HTTPServer(('127.0.0.1', 443), http.server.SimpleHTTPRequestHandler)

	"""
	Convert YouTube subtitles(vtt) to human readable text.

	Download only subtitles from YouTube with youtube-dl:
	youtube-dl --skip-download --convert-subs vtt <video_url>

	Note that default subtitle format provided by YouTube is ass, which is hard
	to process with simple regex. Luckily youtube-dl can convert ass to vtt, which
	is easier to process.

	'''
	Using OpenCV takes a mp4 video and produces a number of images.

	Requirements
	----
	You require OpenCV 3.2 to be installed.

	Run
	----
	Open the main.py and edit the path to the video. Then run: