Tom Cocca tcocca

Sublime Text 2 – Useful Shortcuts (Mac OS X)

Note: these haven't been vetted yet

	#
	# A basic, synthetic benchmark of the impact HTTP client has
	# on the speed of talking to ElasticSearch in the Tire gem.
	#
	# In general, Curb seems to be more then two times faster the RestClient, in some cases it's three
	# to five times faster. I wonder if keep-alive has anything to do with it, but it probably does.
	#
	# Run me with:
	# $ git clone git://github.com/karmi/tire.git
	# $ cd tire

	# Throw this into your spec/support directory
	# Usage:
	#
	# feature 'Dashboard' do
	# scenario 'all' do
	# Given 'I am an authenticated user' do
	# authenticate_user
	# end
	# And 'there are assignments' do
	# create(:assignment)

	require 'restclient'

	# RestClient logs using << which isn't supported by the Rails logger,
	# so wrap it up with a little proxy object.
	RestClient.log =
	Object.new.tap do \|proxy\|
	def proxy.<<(message)
	Rails.logger.info message
	end
	end

	#!/bin/bash
	# herein we backup our indexes! this script should run at like 6pm or something, after logstash
	# rotates to a new ES index and theres no new data coming in to the old one. we grab metadatas,
	# compress the data files, create a restore script, and push it all up to S3.
	TODAY=`date +"%Y.%m.%d"`
	INDEXNAME="logstash-$TODAY" # this had better match the index name in ES
	INDEXDIR="/usr/local/elasticsearch/data/logstash/nodes/0/indices/"
	BACKUPCMD="/usr/local/backupTools/s3cmd --config=/usr/local/backupTools/s3cfg put"
	BACKUPDIR="/mnt/es-backups/"
	YEARMONTH=`date +"%Y-%m"`

	If you want, I can try and help with pointers as to how to improve the indexing speed you get. Its quite easy to really increase it by using some simple guidelines, for example:

	- Use create in the index API (assuming you can).
	- Relax the real time aspect from 1 second to something a bit higher (index.engine.robin.refresh_interval).
	- Increase the indexing buffer size (indices.memory.index_buffer_size), it defaults to the value 10% which is 10% of the heap.
	- Increase the number of dirty operations that trigger automatic flush (so the translog won't get really big, even though its FS based) by setting index.translog.flush_threshold (defaults to 5000).
	- Increase the memory allocated to elasticsearch node. By default its 1g.
	- Start with a lower replica count (even 0), and then once the bulk loading is done, increate it to the value you want it to be using the update_settings API. This will improve things as possibly less shards will be allocated to each machine.
	- Increase the number of machines you have so

	$:.unshift(File.expand_path('../../lib', __FILE__))
	require 'babysitter/puzzle'

	#!/usr/bin/env bash

	###############FUNCTIONS############

	function prepare {
	#optimize the index
	echo -n "Optimizing index $INDEX_NAME..."
	curl -XPOST "$ADDRESS/$INDEX_NAME/_optimize" 2>/dev/null\| grep 'failed":0' >/dev/null
	if [ $? -eq 0 ]; then
	echo "done"

	class TasksController < ApplicationController
	def update
	tracker = TaskTracker.new(@task)
	if @task.update_attributes(params[:task])
	TaskPusher.new(tracker, socket_id).push_changes
	TaskMailSender.new(tracker, current_user).deliver_email
	# success response
	else
	# failure respond
	end