vfdev vfdev-5

(Internal Tranining Material)

Usually the first step in performance optimization is to do profiling, e.g. to identify performance hotspots of a workload. This gist tells basic knowledge of performance profiling on PyTorch, you will get:

How to find the bottleneck operator?
How to trace source file of a particular operator?
How do I indentify threading issues? (oversubscription)
How do I tell a specific operator is running efficiently or not?

This tutorial takes one of my recent projects - pssp-transformer as an example to guide you through path of PyTorch CPU peformance optimization. Focus will be on Part 1 & Part 2.

Polyglot Data Science with IPython & friends

Author: Fernando Pérez.

A demonstration of how to use Python, Julia, Fortran and R cooperatively to analyze data, in the same process.

This is supported by the IPython kernel and a few extensions that take advantage of IPython's magic system to provide low-level integration between Python and other languages.

See the companion notebook for data preparation and setup.

	'''
	To run the CPU benchmark: `CUDA_VISIBLE_DEVICES="" python benchmark.py --name cpu`
	To run the GPU benchmark: `CUDA_VISIBLE_DEVICES=0 python benchmark.py --name cuda`
	To run the distributed benchmark: `python -u -m torch.distributed.launch --nproc_per_node=2 --use_env benchmark.py --name dist`
	'''


	import argparse
	import time
	import math

	#!/bin/bash

	# Activate the build environment
	eval "$(conda shell.bash hook)"
	conda activate pytorch-dev

	cd ~/git/pytorch

	# Enable ccache
	export CCACHE_COMPRESS=true

	import torch
	import warnings
	import threading
	import numpy as np

	from ignite.engine.engine import Engine, Events

	def ensure_tuple(vals):
	"""
	Returns a tuple containing just `vals` if it is not a list or tuple, or `vals` converted to a tuple otherwise.

	# Portions Copyright 2016 The Kubernetes Authors All rights reserved.
	# Portions Copyright 2018 AspenMesh
	#
	# Licensed under the Apache License, Version 2.0 (the "License");
	# you may not use this file except in compliance with the License.
	# You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0
	#
	# Unless required by applicable law or agreed to in writing, software