Adam Duke adamvduke

Internet Scale Services Checklist

A checklist for designing and developing internet scale services, inspired by James Hamilton's 2007 paper "On Desgining and Deploying Internet-Scale Services."

http://mvdirona.com/jrh/talksandpapers/jamesrh_lisa.pdf

Basic tenets

Does the design expect failures to happen regularly and handle them gracefully?
Have we kept things as simple as possible?

Keybase proof

I hereby claim:

I am adamvduke on github.
I am adamvduke (https://keybase.io/adamvduke) on keybase.
I have a public key ASAdtHkKYGody9mlJ2r3_QeHa2uCosnLTuc9xRNrv8fdxwo

To claim this, I am signing this object:

My largest Sidekiq application had a memory leak and I was able to find and fix it in just few hours spent on analyzing Ruby's heap. In this post I'll show my profiling setup.

As you might know Ruby 2.1 introduced a few great changes to ObjectSpace, so now it's much easier to find a line of code that is allocating too many objects. Here is great post explaining how it's working.

I was too lazy to set up some seeding and run it locally, so I checked that test suite passes when profiling is enabled and pushed debugging to production. Production environment also suited me better since my jobs data can't be fully random generated.

So, in order to profile your worker, add this to your Sidekiq configuration:

if ENV["PROFILE"]

	For symmetic encryption, you can use the following:

	To encrypt:
	openssl aes-256-cbc -salt -a -e -in plaintext.txt -out encrypted.txt

	To decrypt:
	openssl aes-256-cbc -salt -a -d -in encrypted.txt -out plaintext.txt


	For Asymmetric encryption you must first generate your private key and extract the public key.

	# If you use bash, this technique isn't really zsh specific. Adapt as needed.
	source ~/keychain-environment-variables.sh

	# AWS configuration example, after doing:
	# $ set-keychain-environment-variable AWS_ACCESS_KEY_ID
	# provide: "AKIAYOURACCESSKEY"
	# $ set-keychain-environment-variable AWS_SECRET_ACCESS_KEY
	# provide: "j1/yoursupersecret/password"
	export AWS_ACCESS_KEY_ID=$(keychain-environment-variable AWS_ACCESS_KEY_ID);
	export AWS_SECRET_ACCESS_KEY=$(keychain-environment-variable AWS_SECRET_ACCESS_KEY);

	example.com {
	root /srv/example.com/public
	tls [email protected]
	proxy / localhost:5000 {
	except /assets
	}
	}

	BEGIN;
	CREATE OR REPLACE FUNCTION primary_streaming_byte_lag() RETURNS TABLE (client_hostname text, client_addr inet, log_location_diff_bytes numeric, total_byte_lag double precision, total_byte_lag_pretty text, replay_byte_lag double precision, replay_byte_lag_pretty text)
	LANGUAGE SQL SECURITY DEFINER
	AS $$
	SELECT
	client_hostname,
	client_addr,
	log_location_diff_bytes,
	( (cur_xlog * 255 * 16 ^ 6) + cur_offset) - ((replay_xlog * 255 * 16 ^ 6) + replay_offset) AS total_byte_lag,
	pg_size_pretty((( (cur_xlog * 255 * 16 ^ 6) + cur_offset) - ((replay_xlog * 255 * 16 ^ 6) + replay_offset))::numeric) AS total_byte_lag_pretty,

	curl -i "https://scontent-a.cdninstagram.com/hphotos-xaf1/outbound-distilleryimage2/t0.0-17/OBPTH/1c8dbe10a8a4....e414f2eeda2_8.jpg"
	HTTP/1.1 400 Not Found
	Content-Type: text/plain
	Server: proxygen
	Timing-Allow-Origin: *
	X-Cache-Ts: 1447545792
	Date: Sun, 15 Nov 2015 00:03:12 GMT
	X-FB-Edge-Debug: BBUHPNJ58swqQlRj0vjMAzLKyzlubMK_sF7iftGX9VpdYAnw9_02fPGxmEeelkNXf_DeSr0oNybV0ztpT-DAjg
	Connection: keep-alive
	Content-Length: 9

	$ dig app5.zeropush.com @ns1.dreamhost.com ANY
	;; Truncated, retrying in TCP mode.

	; <<>> DiG 9.8.3-P1 <<>> app5.zeropush.com @ns1.dreamhost.com ANY
	;; global options: +cmd
	;; Got answer:
	;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62516
	;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
	;; WARNING: recursion requested but not available

	# nginx Configuration File
	# http://wiki.nginx.org/Configuration

	# Run as a less privileged user for security reasons.
	user forge;

	# How many worker threads to run;
	# "auto" sets it to the number of CPU cores available in the system, and
	# offers the best performance. Don't set it higher than the number of CPU
	# cores if changing this parameter.