ifduyue

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

Anonymising PDFs

PDF metadata

Metadata in PDF files can be stored in at least two places:

the Info Dictionary, a limited set of key/value pairs
XMP packets, which contain RDF statements expressed as XML

PDF files

LevelDB is Google's embedded key/value store that uses similar design concepts as the BigTable Tablet

Code

Projects

Routes

小心地使用 Match（Rails 3 已实现）

Rails 3 提供了 match 方法供我们自定义 routes，然而我们要小心使用它以避免“跨站脚本攻击”（XSS Attack）。比如像这样的 routes：

注：（r3 代表 Rails 3，r4 代表 Rails 4）

# routes.rb

The fastest remote directory rsync over ssh archival I can muster (40MB/s over 1gb NICs)

This creates an archive that does the following:

rsync (Everyone seems to like -z, but it is much slower for me)

a: archive mode - rescursive, preserves owner, preserves permissions, preserves modification times, preserves group, copies symlinks as symlinks, preserves device files.
H: preserves hard-links
A: preserves ACLs

	##
	# by SoAwesomeMan
	str =<<-EOS.gsub(/^[\s\t]\|[\s\t]\n/, '') # no space "\s" for new line "\n"; kill tabs too
	select awesome, awesome, awesome, awesome, awesome, awesome,
	from rad, rad, rad, rad, rad, rad, rad, rad, rad, rad, rad,
	where cool cool cool cool cool cool cool cool cool cool cool'
	EOS
	# => "select awesome, awesome, awesome, awesome, awesome, awesome,from rad, rad, rad, rad, rad, rad, rad, rad, rad, rad, rad,where cool cool cool cool cool cool cool cool cool cool cool'"

	str =<<-EOS.gsub(/^[\s\t]/, '').gsub(/[\s\t]\n/, ' ').strip # yes space "\s" for new line "\n"; kill tabs too

	#!/usr/bin/perl



	use Mysql;
	use strict;
	use vars qw($school_name);
	use vars qw($pass);

	require "./cgi-lib.pl";

	###
	### Fast cloning with overlayfs, and specifying user-data on clone
	### blog post:
	### http://ubuntu-smoser.blogspot.com/2013/08/lxc-with-fast-cloning-via-overlayfs-and.html
	###

	### Eventually, this should make it into 13.10 and the stable lxc ppa
	### https://launchpad.net/~ubuntu-lxc/+archive/stable
	### But right now you'll have to use the daily ppa.

	# to generate your dhparam.pem file, run in the terminal
	openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

	local$ socat TCP-LISTEN:2222,bind=127.0.0.1,reuseaddr,fork TCP-LISTEN:2223,reuseaddr
	local$ ssh -p 2222 -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no localhost
	remote$ socat TCP:10.0.2.2:2223 TCP:127.0.0.1:22