kayousterhout

Generating Flame Graphs for Apache Spark

Flame graphs are a nifty debugging tool to determine where CPU time is being spent. Using the Java Flight recorder, you can do this for Java processes without adding significant runtime overhead.

When are flame graphs useful?

Shivaram Venkataraman and I have found these flame recordings to be useful for diagnosing coarse-grained performance problems. We started using them at the suggestion of Josh Rosen, who quickly made one for the Spark scheduler when we were talking to him about why the scheduler caps out at a throughput of a few thousand tasks per second. Josh generated a graph similar to the one below, which illustrates that a significant amount of time is spent in serialization (if you click in the top right hand corner and search for "serialize", you can see that 78.6% of the sampled CPU time was spent in serialization). We used this insight to spee

cocoalabs

man() {
	env \
		LESS_TERMCAP_mb=$(printf "\e[1;31m") \
		LESS_TERMCAP_md=$(printf "\e[1;31m") \
		LESS_TERMCAP_me=$(printf "\e[0m") \
		LESS_TERMCAP_se=$(printf "\e[0m") \
		LESS_TERMCAP_so=$(printf "\e[1;44;33m") \
		LESS_TERMCAP_ue=$(printf "\e[0m") \
		LESS_TERMCAP_us=$(printf "\e[1;32m") \
			man "$@"

Mistobaan

Machine Setup August 2016

Linux Ubuntu 2016.

• 1080 GTX
• SDK 8.0
• CuDNN 5.1

ENABLE Core dumps

ulimit -c unlimited

jarutis

sudo su

# Java
yum -y install java-1.8.0-openjdk-devel

# Build Esentials (minimal)
yum -y install gcc gcc-c++ kernel-devel make automake autoconf swig git unzip libtool binutils

# Extra Packages for Enterprise Linux (EPEL) (for pip, zeromq3)
yum -y install epel-release

marcan

#!/boot/bzImage

# Linux kernel userspace initialization code, translated to bash
# (Minus floppy disk handling, because seriously, it's 2017.)
# Not 100% accurate, but gives you a good idea of how kernel init works
# GPLv2, Copyright 2017 Hector Martin <marcan@marcan.st>
# Based on Linux 4.10-rc2.

# Note: pretend chroot is a builtin and affects the current process
# Note: kernel actually uses major/minor device numbers instead of device name

teamdandelion

7
2
1
0
4
1
4
9
5
9

csswizardry

Vim without NERD tree or CtrlP

I used to use NERD tree for quite a while, then switched to CtrlP for something a little more lightweight. My setup now includes zero file browser or tree view, and instead uses native Vim fuzzy search and auto-directory switching.

Fuzzy Search

There is a super sweet feature in Vim whereby you can fuzzy find your files using **/*, e.g.:

:vs **/*<partial file name><Tab>

aparrish

aparrish

max-mapper