Bill Brower BillBrower

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

AWS ELB with Node.js and WebSockets

This assumes that:

You are using Nginx.
You want to accept incoming connections on port 80.
Your Node.js app is listening on port 3000.
You want to be able to connect to your Node.js instance directly as well as via the load balancer.

####1. Create load balancer

TL;DR

In Ember, always use {{...}}, not {{{...}}}. Use Ember.String.htmlSafe as necessary in JavaScript (usually in a component) to mark markup as HTML-safe. Never pass user-entered content directly to Ember.String.htmlSafe.

Details

Ember has great XSS protection built in. The HTMLBars templating library will automatically run any interpolations through htmlEscape for you. So

(See upstream testem PR: testem/testem#819)

Since around 2012, WebKit and other browsers have throttled timers (like setTimeout) to only a max of once per second in inactive tabs.

For some test suites, this may never be an issue. But, for example, in a very large Ember test suite with >1000 acceptance tests and much custom code, it manifested as often flaky and non-deterministic test timeouts, even in headless CI environments with Chrome in xvfb.

This has caused much much testing pain that might manifest as the not-obviously-related error: Error: timeout of 10000ms exceeded. Ensure the done() callback is being called in this test.

Comprehensions in Python the Jedi way

by Bjørn Friese

Beautiful is better than ugly. Explicit is better than implicit.

-- The Zen of Python

I frequently deal with collections of things in the programs I write. Collections of droids, jedis, planets, lightsabers, starfighters, etc. When programming in Python, these collections of things are usually represented as lists, sets and dictionaries. Oftentimes, what I want to do with collections is to transform them in various ways. Comprehensions is a powerful syntax for doing just that. I use them extensively, and it's one of the things that keep me coming back to Python. Let me show you a few examples of the incredible usefulness of comprehensions.

[![Latest NPM release][npm-badge]][npm-badge-url]
[![TravisCI Build Status][travis-badge-url]][travis-project-url]
[![CircleCI Build Status][circle-badge]][circle-badge-url]
[![Test Coverage][coveralls-badge]][coveralls-badge-url]
[![Code Climate][codeclimate-badge]][codeclimate-badge-url]
[![Ember Observer Score][ember-observer-badge]][ember-observer-badge-url]
[![License][license-badge]][license-badge-url]
[![Dependencies][dependencies-badge]][dependencies-badge-url] 
[![Dev Dependencies][devDependencies-badge]][devDependencies-badge-url]

	Latency Comparison Numbers (~2012)
	----------------------------------
	L1 cache reference 0.5 ns
	Branch mispredict 5 ns
	L2 cache reference 7 ns 14x L1 cache
	Mutex lock/unlock 25 ns
	Main memory reference 100 ns 20x L2 cache, 200x L1 cache
	Compress 1K bytes with Zippy 3,000 ns 3 us
	Send 1K bytes over 1 Gbps network 10,000 ns 10 us
	Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD

	App.InactivityWarningComponent = Ember.Component.extend({

	active: false,
	inactiveTimeout: 12000000, // Amount of time before we redirect to the sign in screen - the session should have expired by this point. (20 minutes)
	warningTimeout: 30000, // Amount of time the user has to perform an action before the last keep alive fires - 30 seconds
	timeout: 1170000, // 19.5 minutes. We want to be less than the 20 minute timeout to be sure the session is renewed.

	didInsertElement: function(){
	//if($('meta[name="in-development"]').attr('content')){ return; } // Uncomment and add a meta tag to your head if you want to avoid session timeout in development
	var context = this;

	def get_count(q):
	count_q = q.statement.with_only_columns([func.count()]).order_by(None)
	count = q.session.execute(count_q).scalar()
	return count

	q = session.query(TestModel).filter(...).order_by(...)

	# Slow: SELECT COUNT(*) FROM (SELECT ... FROM TestModel WHERE ...) ...
	print q.count()

	CREATE TEXT SEARCH CONFIGURATION fr ( COPY = french );
	ALTER TEXT SEARCH CONFIGURATION fr ALTER MAPPING
	FOR hword, hword_part, word WITH unaccent, french_stem;

	CREATE TEXT SEARCH CONFIGURATION en ( COPY = english );
	ALTER TEXT SEARCH CONFIGURATION en ALTER MAPPING
	FOR hword, hword_part, word WITH unaccent, english_stem;

	CREATE TEXT SEARCH CONFIGURATION de ( COPY = german );
	ALTER TEXT SEARCH CONFIGURATION de ALTER MAPPING