Shouda Liu vsooda

A Tour of PyTorch Internals (Part I)

The fundamental unit in PyTorch is the Tensor. This post will serve as an overview for how we implement Tensors in PyTorch, such that the user can interact with it from the Python shell. In particular, we want to answer four main questions:

How does PyTorch extend the Python interpreter to define a Tensor type that can be manipulated from Python code?
How does PyTorch wrap the C libraries that actually define the Tensor's properties and methods?
How does PyTorch cwrap work to generate code for Tensor methods?
How does PyTorch's build system take all of these components to compile and generate a workable application?

Extending the Python Interpreter

PyTorch defines a new package torch. In this post we will consider the ._C module. This module is known as an "extension module" - a Python module written in C. Such modules allow us to define new built-in object types (e.g. the Tensor) and to call C/C++ functions.

	#! /usr/bin/env python2.7
	# coding=utf-8
	""" Trains an agent with (stochastic) Policy Gradients on Pong. Uses OpenAI Gym. """
	import numpy as np
	import cPickle as pickle
	import gym

	# hyperparameters
	H = 200 # number of hidden layer neurons
	batch_size = 10 # every how many episodes to do a param update?

	import pandas as pd
	from random import random

	flow = (list(range(1,10,1)) + list(range(10,1,-1)))*1000
	pdata = pd.DataFrame({"a":flow, "b":flow})
	pdata.b = pdata.b.shift(9)
	data = pdata.iloc[10:] * random() # some noise

	import numpy as np

	"""
	This is a batched LSTM forward and backward pass
	the comment is writen by karpathy, except the comment start with #sooda:
	#sooda: will add some comment corresponding the equtions (ref: lstm.png)
	"""
	import numpy as np
	import code

	class LSTM:

	"""
	Minimal character-level Vanilla RNN model. Written by Andrej Karpathy (@karpathy)
	BSD License
	"""
	import numpy as np

	# data I/O
	data = open('linux_input.txt', 'r').read() # should be simple plain text file
	chars = list(set(data))
	data_size, vocab_size = len(data), len(chars)