Yang Gu greeness

Challenges

query classification: to tell if a text is a review or opinion
which documents or portions of documents contain review-like or opinionated material.
identifying the overall sentiment expressed
the system needs to present the sentiment information it has garnered in some reasonable summary fashion.
aggregation of "votes" that may re registered on different scales

	# launch an ec2 instance with lucid (ubuntu 10.04) e.g. ami-ad36fbc4
	# ssh to the machine

	################################################################
	# install java
	# https://ccp.cloudera.com/display/CDHDOC/Java+Development+Kit+Installation
	# RELEASE=lucid, which you can find by running lsb_release -c.
	################################################################

	$ sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"

	dumbo start demo_dumbo.py -hadoop /usr/lib/hadoop -input shares -output video_demos -outputformat text -files hdfs://ec2-xxx-xx-xx-xx.compute-1.amazonaws.com:8020/user/ubuntu/users/part-m-00000

	### piece of code in demo_dumbo.py
	for line in file('part-m-00000'):
	print line

	# ----------------
	dumbo start demo_dumbo.py -hadoop /usr/lib/hadoop -input shares -output video_demos -outputformat text -files hdfs://ec2-xxx-xx-xx-xx.compute-1.amazonaws.com:8020/user/ubuntu/users

	sudo apt-get install gfortran libblas-dev liblapack-dev

	cd scipy-0.10.0
	python setupegg.py bdist_egg

	# wait about 5 minutes on my machine

	import numpy
	import math

	# LSH signature generation using random projection
	def get_signature(user_vector, rand_proj):
	res = 0
	for p in (rand_proj):
	res = res << 1
	val = numpy.dot(p, user_vector)
	if val >= 0:

	http://mrtopf.de/blog/en/a-small-introduction-to-python-eggs/

	sudo apt-get install python-setuptools

	python setup.py bdist_egg

	npm install

	nodeunit tests/recommender_tests.coffee