Zhengren Pan zhp007

Introduction

An introduction to curl using GitHub's API.

The Basics

Makes a basic GET request to the specifed URI

curl https://api.github.com/users/caspyin

Linux simple performance tweaks

Change the I/O Scheduler

Open $ vim /etc/default/grub then add elevator=noop next to GRUB_CMDLINE_LINUX_DEFAULT. Run $ update-grub and $ cat /sys/block/sda/queue/scheduler to be sure that noop is being used:

$ vim /etc/default/grub
$ update-grub
$ cat /sys/block/sda/queue/scheduler

[noop] deadline cfq

Mac OSX

LZO

brew install lzop lzo

Hadoop

Compile Hadoop LZO library:

Memory Optimization (Christer Ericson, GDC 2003)
http://realtimecollisiondetection.net/pubs/GDC03_Ericson_Memory_Optimization.ppt

Cache coherency primer (Fabian Giesen)
https://fgiesen.wordpress.com/2014/07/07/cache-coherency/

Code Clinic 2015: How to Write Code the Compiler Can Actually Optimize (Mike Acton)
http://gdcvault.com/play/1021866/Code-Clinic-2015-How-to

These are the Kickstarter Engineering and Data role definitions for both teams.

Technical	Data	People
Junior Software Engineer
Software Engineer	Data Analyst
Senior Software Engineer	Data Scientist	Engineering Manager
Staff Engineer	VP of Data	Engineering Director
Principal Engineer		CTO

Common Options

-#, --progress-bar Make curl display a simple progress bar instead of the more informational standard meter.

-b, --cookie <name=data> Supply cookie with request. If no =, then specifies the cookie file to use (see -c).

-c, --cookie-jar <file name> File to save response cookies to.

Enable native memory tracking in JVM by specifying the following flag

-XX:NativeMemoryTracking=detail

Know the <PID> of the java process

jps

To print ps based RSS

ps -p <PID> -o pcpu,rss,size,vsize

To print native memory tracking summary

All Your Data Does Not Belong In a Database

Businesses are machines producing mountains of data about sales, usage, customer, costs, etc... Traditionally data processing is highly centralised with teams of staff and computer running hot a whirling ready to process. We can do better than moving the mountain of data into the corporate data machine - so long as that machinary is light enough to be moved to the data.

Don't move the mountain - Bring the processing to the data

We've had this problem; a huge directory of files in CSV format, conataining vital information for our business. But it's in CSV, requires analysis, and don't you don't feel like learning sed/grep/awk today - besides it's 2017 and no-one thinks those tools are easy to use.

 describe vs. context in rspec
 http://lmws.net/describe-vs-context-in-rspec
 In Rspec world, you often see people using both “describe” blocks and “context” blocks together, like this
 describe "launch the rocket" do
   context "all ready" do
   end
   context "not ready" do

	# You don't need Fog in Ruby or some other library to upload to S3 -- shell works perfectly fine
	# This is how I upload my new Sol Trader builds (http://soltrader.net)
	# Based on a modified script from here: http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

	S3KEY="my aws key"
	S3SECRET="my aws secret" # pass these in

	function putS3
	{
	path=$1