Silviu Caragea silviucpp

Using OTP's observer (appmon replacement) remotely

$ ssh remote-host "epmd -names"
epmd: up and running on port 4369 with data:
name some_node at port 58769

Note the running on port for epmd itself and the port of the node you're interested in debugging. Reconnect to the remote host with these ports forwarded:

$ ssh -L 4369:localhost:4369 -L 58769:localhost:58769 remote-host

Scaling your API with rate limiters

The following are examples of the four types rate limiters discussed in the accompanying blog post. In the examples below I've used pseudocode-like Ruby, so if you're unfamiliar with Ruby you should be able to easily translate this approach to other languages. Complete examples in Ruby are also provided later in this gist.

In most cases you'll want all these examples to be classes, but I've used simple functions here to keep the code samples brief.

Request rate limiter

This uses a basic token bucket algorithm and relies on the fact that Redis scripts execute atomically. No other operations can run between fetching the count and writing the new count.

Installing vitess

First of we need the registry and mysql server installed so we can keep track on which services are available, download new services and also run the actual database.

sudo apt install -y default-mysql-server default-mysql-client etcd curl

Before we start this POC we disable the default services as we will be setting up our own.

Better Primary Keys / PK / id

Why not just use sequential ID's (`SERIAL` or `AUTO INCREMENT`) ?

Scaling writes require partitions / shards at some level.
- Avoid SERIAL and AUTO INCREMENT from start.
  - Servers have to ask "what's the next available id in the sequence?" = huge global contention / slow down.
  - Turns into a queue. OK for 1 server. Bad for many servers.
    - Vitess and FoundationDB can work around this using sequence counters per server. Vitess one has an awful DX. Not sure about FoundationDB.
- Alternative: Use time or random.
UUID bonus: Split, merge tables, and load any rows any time you wish, since rows are globally unique.

	#! /bin/sh
	### BEGIN INIT INFO
	# Provides: supervisord
	# Required-Start: $remote_fs
	# Required-Stop: $remote_fs
	# Default-Start: 2 3 4 5
	# Default-Stop: 0 1 6
	# Short-Description: Example initscript
	# Description: This file should be used to construct scripts to be
	# placed in /etc/init.d.

	<?php

	include 'vendor/autoload.php';

	$pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1');
	$pheanstalk->useTube('default');

	$n = 1000000;
	$start = microtime(true);

	0815.ru
	0815.ru0clickemail.com
	0815.ry
	0815.su
	0845.ru
	0clickemail.com
	0-mail.com
	0wnd.net
	0wnd.org
	10mail.com

	:) SELECT toDateTime(intDiv(toUInt32(EventTime), 5) * 5) AS k, count(), uniq(UserID) FROM hits_layer WHERE CounterID = 29761725 AND EventDate = today() AND EventTime >= now() - 600 GROUP BY k ORDER BY k

	SELECT
	toDateTime(intDiv(toUInt32(EventTime), 5) * 5) AS k,
	count(),
	uniq(UserID)
	FROM hits_layer
	WHERE (CounterID = 29761725) AND (EventDate = today()) AND (EventTime >= (now() - 600))
	GROUP BY k
	ORDER BY k ASC