mt3 mt3

Here are the areas I've been researching, some things I've read and some open source packages...

Nearly all text processing starts by transforming text into vectors: http://en.wikipedia.org/wiki/Vector_space_model

Often it uses transforms such as TFIDF to normalise the data and control for outliers (words that are too frequent or too rare confuse the algorithms): http://en.wikipedia.org/wiki/Tf%E2%80%93idf

Collocations is a technique to detect when two or more words occur more commonly together than separately (e.g. "wishy-washy" in English) - I use this to group words into n-gram tokens because many NLP techniques consider each word as if it's independent of all the others in a document, ignoring order: http://matpalm.com/blog/2011/10/22/collocations_1/

Yesterday I upgraded our running elasticsearch cluster on a site which serves a few million search requests a day, with zero downtime. I've been asked to describe the process, hence this blogpost.

To make it more complicated, the cluster was running elasticsearch version 0.17.8 (released 6 Oct 2011) and I upgraded it to the latest 0.19.10. There have been 21 releases between those two versions, with a lot of functional changes, so I needed to be ready to roll back if necessary.

Our setup:

elasticsearch

We run elasticsearch on two biggish boxes: 16 cores plus 32GB of RAM. All indices have 1 replica, so all data is stored on both boxes (about 45GB of data). The primary data for our main indices is also stored in our database. We have a few other indices whose data is stored only in elasticsearch, but are updated once daily only. Finally, we store our sessions in elasticsearch, but active sessions are cached in memcached.

	import urllib2
	import json

	import pprint
	import datetime
	pp = pprint.PrettyPrinter(indent=4)



	# Get Date

	from scene import *
	#from random import random
	from time import ctime
	from os import listdir, path, stat
	from os.path import isdir, isfile, sep, abspath
	interesting_dirs = {0:'.', 1:'/',2:'/Applications',3: '/Applications/Maps.app'}
	#t= '/Applications/Maps.app/Japanese.lproj/Localizable.strings'

	def dir_contents(path):
	"""get all dirs and files from path"""

	from fabric.context_managers import cd
	from fabric.contrib.files import exists
	from fabric.decorators import hosts
	from fabric.decorators import task
	from fabric.operations import run
	from fabric.operations import sudo

	PACKAGES = [
	# 'python-mapnik',
	'binutils',

	--
	-- Created by: Sean Korzdorfer
	-- Created on: 06/08/12 17:05:27
	-- This script incudes code from an example posted by Rob Trew
	-- String Library can be obtained at:
	-- http://applescript.bratis-lover.net/library/string/
	-- Any line which contains a path needs to be edited for the users environment
	-- NB: this could easily be edited to append.

	# File Transfer for Pythonista
	# ============================
	# This script allows you to transfer Python files from
	# and to Pythonista via local Wifi.
	# It starts a basic HTTP server that you can access
	# as a web page from your browser.
	# When you upload a file that already exists, it is
	# renamed automatically.
	# From Pythonista's settings, you can add this script
	# to the actions menu of the editor for quick access.

	#!/usr/bin/env python

	import sys
	from urllib import urlencode

	param = '%clipboard'.strip()
	rooturl = "http://lmgtfy.com/?"
	try:
	qs = urlencode({'q':param})
	except:

	"""
	Copyright (c) 2012 Zohaib Sibte Hassan

	Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

	The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR