Lishi He ikbear

A friend asked me for a few pointers to interesting, mostly recent papers on data warehousing and "big data" database systems, with an eye towards real-world deployments. I figured I'd share the list. While it's biased and rather incomplete but maybe of interest to someone. While many are obvious choices (I've omitted several, like MapReduce), I think there are a few underappreciated gems.

###Dataflow Engines:

Dryad--general-purpose distributed parallel dataflow engine
http://research.microsoft.com/en-us/projects/dryad/eurosys07.pdf

Spark--in memory dataflow
http://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdf

下面是陈首席陈临怀发表在微博上的 [NetScreen的往事]，我在看的过程中将其拷贝出来，整理成文，方便想看的人一口气看完。微博见：http://weibo.com/huailinchen

［NetScreen的往事］下午要在厦门见邓锋。突然又想起了许多NetScreen的往事。我是一个能写代码的文艺青年。决定在weibo上，来写写过去，算作为逐渐老去的我们的一点纪念。。。

［NetScreen的往事（1）］2001年的9月11日的早晨。我大概7点左右被人叫醒。纽约遭遇攻击！“。在电视上我看着第2架飞机冲入Tower Building，恍惚做梦一般。。。上午去公司。。。大家都很担心。知道NetScreen的创办人Feng和Yan也在纽约，处理公司上市的事情。大家都很担心他们。我们是兄弟。

［NetScreen的往事（2）］应该是中午的时候传来消息。他们已经跑到亚特兰大去了。在等待转机。。。大家都在等着他们的回来。记得是晚上7点左右，夏天的加州，依然天很亮。我和Shalang（现Juniper Beijing Director）在楼下抽烟。看见Feng和Yan下车了。。。我是个很感情化的人，冲上去拥抱了Feng。

［NetScreen的往事（3）］Yan是个比较严肃的人，而我的工作基本上是他和Yuming［Palo Alto创办人，今年上市，发大了］直接领导。他经常修理我。有点怕他。不敢和他拥抱。Feng很随和，喜欢开玩笑［含黄色段子］。大家都喜欢他。。。

	#!/usr/bin/env ruby
	require 'thor'
	require 'rest_client'
	require 'base64'
	require 'qiniu-rs'

	class QiniuUploader < Thor
	desc "upload", "Upload a file to qiniu cloud storage"
	def upload
	bucket = 'mybucket'

	package main

	// sort a map's keys in descending order of its values.

	import "sort"

	type sortedMap struct {
	m map[string]int
	s []string
	}

	require "nokogiri"
	require "open-uri"
	doc = Nokogiri::HTML(open("http://vladimirzotov.500px.com/").read)
	imgs = doc.css('img').map { \|i\| i['src'] }

	i = 0
	for img in imgs
	data = open(img) { \|f\| f.read }
	open(i.to_s + ".jpg", "wb") { \|f\| f.write(data) }
	i += 1

	#!/usr/bin/env python
	# encoding: utf-8

	import sys

	def main():
	f = open('data.txt')
	c1, c2, c3 = 0.0, 0.0, 0.0
	lines = f.readlines()
	length = len(lines)

	#!/bin/bash
	# encoding: utf-8

	rm data.txt
	i=1
	while [ "$i" -le 100 ]
	do
	python sim.py
	i=$((i+1))
	done

	#!/usr/bin/env python
	# encoding: utf-8
	import random

	# Parameters: N for N clusters,
	# P for the computing power of a cluster,
	# M for M requests at epoch t,
	# T for the teardown time of a request.
	# Qmin, Qavg, Qmax are for probability
	# distribution law II(Qmin, Qavg, Qmax).

	require 'fileutils'
	require 'date'
	require 'yaml'
	require 'rexml/document'
	require 'ya2yaml'

	include REXML

	doc = Document.new(File.new(ARGV[0]))

	# Gemfile
	gem 'redis', '~> 3.0.1'