Peter Ajtai pajtai

Opening and closing an SSH tunnel in a shell script the smart way

I recently had the following problem:

From an unattended shell script (called by Jenkins), run a command-line tool that accesses the MySQL database on another host.
That tool doesn't know that the database is on another host, plus the MySQL port on that host is firewalled and not accessible from other machines.

We didn't want to open the MySQL port to the network, but it's possible to SSH from the Jenkins machine to the MySQL machine. So, basically you would do something like

ssh -L 3306:localhost:3306 remotehost

Turning PostgreSQL into a queue serving 10,000 jobs per second

RDBMS-based job queues have been criticized recently for being unable to handle heavy loads. And they deserve it, to some extent, because the queries used to safely lock a job have been pretty hairy. SELECT FOR UPDATE followed by an UPDATE works fine at first, but then you add more workers, and each is trying to SELECT FOR UPDATE the same row (and maybe throwing NOWAIT in there, then catching the errors and retrying), and things slow down.

On top of that, they have to actually update the row to mark it as locked, so the rest of your workers are sitting there waiting while one of them propagates its lock to disk (and the disks of however many servers you're replicating to). QueueClassic got some mileage out of the novel idea of randomly picking a row near the front of the queue to lock, but I can't still seem to get more than an an extra few hundred jobs per second out of it under heavy load.

So, many developers have started going straight t

What I actually did

/etc/security/limits.conf

elasticsearch hard memlock 100000

/etc/default/elasticsearch

	#!/bin/bash
	### BEGIN INIT INFO
	# Provides: kibana
	# Required-Start: $remote_fs $syslog
	# Required-Stop: $remote_fs $syslog
	# Default-Start: 2 3 4 5
	# Default-Stop: 0 1 6
	# Short-Description: Make sense of a mountain of logs.
	### END INIT INFO

	(function() {
	var CSSCriticalPath = function(w, d, opts) {
	var opt = opts \|\| {};
	var css = {};
	var pushCSS = function(r) {
	if(!!css[r.selectorText] === false) css[r.selectorText] = {};
	var styles = r.style.cssText.split(/;(?![A-Za-z0-9])/);
	for(var i = 0; i < styles.length; i++) {
	if(!!styles[i] === false) continue;
	var pair = styles[i].split(": ");

	document.getElementsByTagName('button')[0].onclick = function () {
	scrollTo(document.body, 0, 1250);
	}

	function scrollTo(element, to, duration) {
	var start = element.scrollTop,
	change = to - start,
	currentTime = 0,
	increment = 20;

	<VirtualHost *:{PORT}>

	ServerName www.yourdomain.com
	ServerAdmin [email protected]
	DocumentRoot /var/www/yourdir/

	<Directory /var/www/yourdir>
	Options Indexes FollowSymLinks
	AllowOverride all
	Order allow,deny

	# Feel free to change this path of course (and keys_zone value as well, but also change the usage of it below).
	proxy_cache_path /var/www/cache/resized levels=1:2 keys_zone=resizedimages:10m max_size=1G;

	# Gzip was on in another conf file of mine...You may need to uncomment the next line.
	#gzip on;
	gzip_disable msie6;
	gzip_static on;
	gzip_comp_level 4;
	gzip_proxied any;
	# Again, be careful that you aren't overwriting some other setting from another config's http {} section.

	# to generate your dhparam.pem file, run in the terminal
	openssl dhparam -out /etc/nginx/ssl/dhparam.pem 2048

	App configuration in environment variables: for and against

	For (some of these as per the 12 factor principles)

	1) they are are easy to change between deploys without changing any code

	2) unlike config files, there is little chance of them being checked
	into the code repo accidentally

	3) unlike custom config files, or other config mechanisms such as Java