Oleg Kachan oleg-kachan

High-Performance Matrix Multiplication

This is a short post that explains how to write a high-performance matrix multiplication program on modern processors. In this tutorial I will use a single core of the Skylake-client CPU with AVX2, but the principles in this post also apply to other processors with different instruction sets (such as AVX512).

Intro

Matrix multiplication is a mathematical operation that defines the product of

A Tour of PyTorch Internals (Part I)

The fundamental unit in PyTorch is the Tensor. This post will serve as an overview for how we implement Tensors in PyTorch, such that the user can interact with it from the Python shell. In particular, we want to answer four main questions:

How does PyTorch extend the Python interpreter to define a Tensor type that can be manipulated from Python code?
How does PyTorch wrap the C libraries that actually define the Tensor's properties and methods?
How does PyTorch cwrap work to generate code for Tensor methods?
How does PyTorch's build system take all of these components to compile and generate a workable application?

Extending the Python Interpreter

PyTorch defines a new package torch. In this post we will consider the ._C module. This module is known as an "extension module" - a Python module written in C. Such modules allow us to define new built-in object types (e.g. the Tensor) and to call C/C++ functions.

Curriculum Learning

Introduction

Curriculum Learning - When training machine learning models, start with easier subtasks and gradually increase the difficulty level of the tasks.
Motivation comes from the observation that humans and animals seem to learn better when trained with a curriculum like a strategy.
Link to the paper.

Contributions of the paper

Quick List of Resources for Topological Data Analysis with Emphasis on Machine Learning

This is just a quick list of resourses on TDA that I put together for @rickasaurus after he was asking for links to papers, books, etc on Twitter and is by no means an exhaustive list.

Survey Papers

Both Carlsson's and Ghrist's survey papers offer a very good introduction to the subject

Topology and Data by Gunnar Carlsson
Barcodes: The Persistent Topology of Data by Robert Ghrist

Other Papers and Web Resources

Extracting insights from the shape of complex data using topology A good introductory paper in Nature on the Mapper algorithm.

OpenCV Video Writer on Mac OS X

Fourcc tests:

x : not worked
o: worked (able to open in quicktime but need conversion)
oo: worked well (preview worked but wrong aspect ratio)
ooo: worked very well (preview worked, and aspect ratio’s right)

(Codecs are extracted from https://web.archive.org/web/20120722124832/http://opencv.willowgarage.com/wiki/QuickTimeCodecs )

NOTE: This was first authored on 26 Feb 2014. Things may have changed since then.

C++'s Templates

C++'s templates could be seen as forming a duck typed, purely functional code generation program that is run at compile time. Types are not checked at the initial invocation stage, rather the template continues to expand until it is either successful, or runs into an operation that is not supported by that specific type – in that case the compiler spits out a 'stack trace' of the state of the template expansion.

To see this in action, lets look at a very simple example:

template

	def mmul(*tensors):
	return tf.foldl(tf.matmul, tensors)

	def msym(X):
	return (X + tf.matrix_transpose(X)) / 2

	def mdiag(X):
	return tf.matrix_diag(tf.matrix_diag_part(X))

	@tf.RegisterGradient('Svd')

	# Drawn from https://gist.github.com/rocknrollnerd/c5af642cf217971d93f499e8f70fcb72 (in Theano)
	# This is implemented in PyTorch
	# Author : Anirudh Vemula

	import torch
	import torch.nn as nn
	from torch.autograd import Variable
	import numpy as np

	from sklearn.datasets import fetch_mldata