nerguri nerguri

Machine Setup August 2016

Linux Ubuntu 2016.

1080 GTX
SDK 8.0
CuDNN 5.1

ENABLE Core dumps

ulimit -c unlimited

Generating Flame Graphs for Apache Spark

Flame graphs are a nifty debugging tool to determine where CPU time is being spent. Using the Java Flight recorder, you can do this for Java processes without adding significant runtime overhead.

When are flame graphs useful?

Shivaram Venkataraman and I have found these flame recordings to be useful for diagnosing coarse-grained performance problems. We started using them at the suggestion of Josh Rosen, who quickly made one for the Spark scheduler when we were talking to him about why the scheduler caps out at a throughput of a few thousand tasks per second. Josh generated a graph similar to the one below, which illustrates that a significant amount of time is spent in serialization (if you click in the top right hand corner and search for "serialize", you can see that 78.6% of the sampled CPU time was spent in serialization). We used this insight to spee

Automation for the People

Long ago, the first time I read "The Pragmatic Programmer", I read some advice that really stuck with me.

"Don't Use Manual Procedures".

This in the chapter on Ubiquitous Automation. To summarize, they want you to automate all the things.

The trouble was that I hadn't much of an idea how to actually go

Installing the Oracle Java 8 JDK on SmartOS

Go to: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Download the "Solaris x64 XX.X MB jdk-8u66-solaris-x64.tar.gz" tarball from the Oracle Java SE site. You won't be able to paste the URL into curl on your SmartOS instance unless you click it first to get the authentication parameter. Regardless, get the tarball any way that you prefer and copy it onto your SmartOS instance.
Extract the tarball and copy it to the location of your choosing.
Globally set the value of the environment variable JAVA_HOME to the path of the JVM.
Update your PATH to include the Java bin directory by setting it to PATH=$PATH:$JAVA_HOME/bin

Memory Optimization (Christer Ericson, GDC 2003)
http://realtimecollisiondetection.net/pubs/GDC03_Ericson_Memory_Optimization.ppt

Cache coherency primer (Fabian Giesen)
https://fgiesen.wordpress.com/2014/07/07/cache-coherency/

Code Clinic 2015: How to Write Code the Compiler Can Actually Optimize (Mike Acton)
http://gdcvault.com/play/1021866/Code-Clinic-2015-How-to

	sudo su

	# Java
	yum -y install java-1.8.0-openjdk-devel

	# Build Esentials (minimal)
	yum -y install gcc gcc-c++ kernel-devel make automake autoconf swig git unzip libtool binutils

	# Extra Packages for Enterprise Linux (EPEL) (for pip, zeromq3)
	yum -y install epel-release

	man() {
	env \
	LESS_TERMCAP_mb=$(printf "\e[1;31m") \
	LESS_TERMCAP_md=$(printf "\e[1;31m") \
	LESS_TERMCAP_me=$(printf "\e[0m") \
	LESS_TERMCAP_se=$(printf "\e[0m") \
	LESS_TERMCAP_so=$(printf "\e[1;44;33m") \
	LESS_TERMCAP_ue=$(printf "\e[0m") \
	LESS_TERMCAP_us=$(printf "\e[1;32m") \
	man "$@"

	# Add this snippet to the top of your playbook.
	# It will install python2 if missing (but checks first so no expensive repeated apt updates)
	# gwillem@gmail.com

	- hosts: all
	gather_facts: False

	tasks:
	- name: install python 2
	raw: test -e /usr/bin/python \|\| (apt -y update && apt install -y python-minimal)

	{0: 'tench, Tinca tinca',
	1: 'goldfish, Carassius auratus',
	2: 'great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias',
	3: 'tiger shark, Galeocerdo cuvieri',
	4: 'hammerhead, hammerhead shark',
	5: 'electric ray, crampfish, numbfish, torpedo',
	6: 'stingray',
	7: 'cock',
	8: 'hen',
	9: 'ostrich, Struthio camelus',

	global
	log /dev/log local0
	log /dev/log local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon
	maxconn 2048