vfdev vfdev-5

Polyglot Data Science with IPython & friends

A demonstration of how to use Python, Julia, Fortran and R cooperatively to analyze data, in the same process.

This is supported by the IPython kernel and a few extensions that take advantage of IPython's magic system to provide low-level integration between Python and other languages.

See the companion notebook for data preparation and setup.

(Internal Tranining Material)

Usually the first step in performance optimization is to do profiling, e.g. to identify performance hotspots of a workload. This gist tells basic knowledge of performance profiling on PyTorch, you will get:

How to find the bottleneck operator?
How to trace source file of a particular operator?
How do I indentify threading issues? (oversubscription)
How do I tell a specific operator is running efficiently or not?

This tutorial takes one of my recent projects - pssp-transformer as an example to guide you through path of PyTorch CPU peformance optimization. Focus will be on Part 1 & Part 2.

	# Portions Copyright 2016 The Kubernetes Authors All rights reserved.
	# Portions Copyright 2018 AspenMesh
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software

	import torch
	import warnings
	import threading
	import numpy as np

	from ignite.engine.engine import Engine, Events

	def ensure_tuple(vals):
	"""
	Returns a tuple containing just `vals` if it is not a list or tuple, or `vals` converted to a tuple otherwise.

	#!/bin/bash

	# Activate the build environment
	eval "$(conda shell.bash hook)"
	conda activate pytorch-dev

	cd ~/git/pytorch

	# Enable ccache
	export CCACHE_COMPRESS=true

	'''
	To run the CPU benchmark: `CUDA_VISIBLE_DEVICES="" python benchmark.py --name cpu`
	To run the GPU benchmark: `CUDA_VISIBLE_DEVICES=0 python benchmark.py --name cuda`
	To run the distributed benchmark: `python -u -m torch.distributed.launch --nproc_per_node=2 --use_env benchmark.py --name dist`
	'''


	import argparse
	import time
	import math