Skip to content

Instantly share code, notes, and snippets.

@chanks
chanks / gist:7585810
Last active September 17, 2024 11:55
Turning PostgreSQL into a queue serving 10,000 jobs per second

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

@alexjs
alexjs / cors-nginx.conf
Created November 28, 2012 22:42 — forked from michiel/cors-nginx.conf
Slightly tighter CORS config for nginx
#
# Slightly tighter CORS config for nginx
#
# A modification of https://gist.github.com/1064640/ to include a white-list of URLs
#
# Despite the W3C guidance suggesting that a list of origins can be passed as part of
# Access-Control-Allow-Origin headers, several browsers (well, at least Firefox)
# don't seem to play nicely with this.
#
@piscisaureus
piscisaureus / pr.md
Created August 13, 2012 16:12
Checkout github pull requests locally

Locate the section for your github remote in the .git/config file. It looks like this:

[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = [email protected]:joyent/node.git

Now add the line fetch = +refs/pull/*/head:refs/remotes/origin/pr/* to this section. Obviously, change the github url to match your project's URL. It ends up looking like this:

@zhengjia
zhengjia / capybara cheat sheet
Created June 7, 2010 01:35
capybara cheat sheet
=Navigating=
visit('/projects')
visit(post_comments_path(post))
=Clicking links and buttons=
click_link('id-of-link')
click_link('Link Text')
click_button('Save')
click('Link Text') # Click either a link or a button
click('Button Value')