You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Instantly share code, notes, and snippets.
👷♂️
Building
Daniel Schniepp
dschniepp
👷♂️
Building
I am passionate about empowering people and creating software based products - People, Product, Code
Turning PostgreSQL into a queue serving 10,000 jobs per second
Turning PostgreSQL into a queue serving 10,000 jobs per second
RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.
On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On choosing a hypermedia type for your API - HAL, JSON-LD, Collection+JSON, SIREN, Oh My!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When hosting our web applications, we often have one public IP
address (i.e., an IP address visible to the outside world)
using which we want to host multiple web apps. For example, one
may wants to host three different web apps respectively for
example1.com, example2.com, and example1.com/images on
the same machine using a single IP address.
How can we do that? Well, the good news is Internet browsers
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Using OTP's observer (appmon replacement) remotely
$ ssh remote-host "epmd -names"
epmd: up and running on port 4369 with data:
name some_node at port 58769
Note the running on port for epmd itself and the port of the node you're interested in debugging. Reconnect to the remote host with these ports forwarded:
Build, test, cache and deploy docker images with wercker, quay.io and Elastic Beanstalk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AWS: CloudFormation with Elastic Beanstalk, Docker Go web app.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters