mt3 mt3

Yesterday I upgraded our running elasticsearch cluster on a site which serves a few million search requests a day, with zero downtime. I've been asked to describe the process, hence this blogpost.

To make it more complicated, the cluster was running elasticsearch version 0.17.8 (released 6 Oct 2011) and I upgraded it to the latest 0.19.10. There have been 21 releases between those two versions, with a lot of functional changes, so I needed to be ready to roll back if necessary.

Our setup:

elasticsearch

We run elasticsearch on two biggish boxes: 16 cores plus 32GB of RAM. All indices have 1 replica, so all data is stored on both boxes (about 45GB of data). The primary data for our main indices is also stored in our database. We have a few other indices whose data is stored only in elasticsearch, but are updated once daily only. Finally, we store our sessions in elasticsearch, but active sessions are cached in memcached.

	import subprocess
	import os
	import time

	from watchdog.observers import Observer
	from watchdog.events import FileSystemEventHandler


	class ChangeHandler(FileSystemEventHandler):
	"""React to modified .tex files."""

	# Checkout remote branch
	git checkout -t origin/<branch>

	# Delete branch (local, remote)
	git branch -d <branch>
	git push origin --delete <branch>

	# Submodules
	git submodule init
	git submodule update

	// Stripped down jQuerify for modern browsers: http://www.learningjquery.com/2009/04/better-stronger-safer-jquerify-bookmarklet

	(function() {
	var f = function() {
	jQuery.noConflict()(function($) {
	// Bookmarklet code goes here
	});
	};

	if (typeof jQuery != 'undefined') {

	in the: ************************************* (3765)
	i love: *************************** (2781)
	to be: ************************** (2637)
	of the: ********************* (2150)
	if you: ********************* (2150)
	on the: ******************** (2081)
	i just: ****************** (1838)
	i was: ****************** (1829)
	i don't: ****************** (1825)
	i have: ****************** (1802)

	var casper = require("casper").create();
	var url = "https://github.com/search?langOverride=VimL&language=&q=vimrc&repo=&type=Repositories&start_value=" + casper.cli.get(0);
	var repoLinks = [];
	var fileLinks = [];

	function getRepoLinks() {
	var links = [];
	$("div.results .result h2 a").each(function(i,el){
	links.push(el.href);
	});

	# Credit http://stackoverflow.com/a/2514279
	for branch in `git branch -r \| grep -v HEAD`;do echo -e `git show --format="%ci %cr" $branch \| head -n 1` \\t$branch; done \| sort -r

	# -- coding: utf-8 --

	import os

	from flask import Flask

	from flask_heroku import Heroku
	from flask_sslify import SSLify
	from raven.contrib.flask import Sentry
	from flask.ext.celery import Celery

	\documentclass$if(fontsize)$[$fontsize$]$endif${article}
	\usepackage{amssymb,amsmath}
	\usepackage{ifxetex,ifluatex}
	\ifxetex
	\usepackage{fontspec,xltxtra,xunicode}
	\defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}
	\else
	\ifluatex
	\usepackage{fontspec}
	\defaultfontfeatures{Mapping=tex-text,Scale=MatchLowercase}

	.DS_Store
	*.log
	Gemfile.lock