Marek Wywiał onjin

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Optimizing postgresql table for more than 100K inserts per second

Create UNLOGGED table. This reduces the amount of data written to persistent storage by up to 2x.
Set WITH (autovacuum_enabled=false) on the table. This saves CPU time and IO bandwidth on useless vacuuming of the table (since we never DELETE or UPDATE the table).
Insert rows with COPY FROM STDIN. This is the fastest possible approach to insert rows into table.
Minimize the number of indexes in the table, since they slow down inserts. Usually an index on time timestamp with time zone is enough.
Add synchronous_commit = off to postgresql.conf.
Use table inheritance for fast removal of old data:

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	-- show running queries (pre 9.2)
	SELECT procpid, age(clock_timestamp(), query_start), usename, current_query
	FROM pg_stat_activity
	WHERE current_query != '<IDLE>' AND current_query NOT ILIKE '%pg_stat_activity%'
	ORDER BY query_start desc;

	-- show running queries (9.2)
	SELECT pid, age(clock_timestamp(), query_start), usename, query
	FROM pg_stat_activity
	WHERE query != '<IDLE>' AND query NOT ILIKE '%pg_stat_activity%'

	# coding: utf-8
	# copyright 2014 tshirtman
	# distributed under the GPL licence

	from __future__ import division

	from kivy.app import App
	from kivy.lang import Builder
	from kivy.properties import ListProperty, NumericProperty, StringProperty
	from kivy.clock import Clock

	#!/usr/bin/env python3
	# encoding: utf-8

	"""
	I saw a similar script on the homepage of Miguel Grinberg (the Flask book guy),
	but he was using webscraping. Here I use simple API calls instead.

	The script takes the presentations of a Python conference and orders the
	presentations in descending order by the number of youtube views. It
	is an indicator about the popularity of a video.

	from abc import ABC, ABCMeta, abstractmethod
	from collections import namedtuple
	from itertools import count


	PayloadFactory = namedtuple('PayloadFactory', [
	'good', 'created', 'queued', 'unchanged', 'requires_auth',
	'permission_denied', 'not_found', 'invalid', 'error'
	])
	"""

	# to generate your dhparam.pem file, run in the terminal
	openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

	class Specification:

	def __and__(self, other):
	return And(self, other)

	def __or__(self, other):
	return Or(self, other)

	def __xor__(self, other):
	return Xor(self, other)

	import decimal
	from functools import total_ordering
	from numbers import Real

	class Context(object):
	def __init__(self, **kwargs):
	self.context = decimal.Context(**kwargs)

	def __enter__(self):
	with decimal.localcontext(self.context) as c: