heapwolf heapwolf

build

Clone and build Node for analysis:

$ git clone https://github.com/joyent/node.git
$ cd node
$ export GYP_DEFINES="v8_enable_disassembler=1 v8_object_print=1"
$ export CXXFLAGS="-fno-omit-frame-pointer"
$ ./configure

Synopsis

In Pull-Streams, there are two fundamental types of streams Sources and Sinks. There are two composite types of streams Through (aka transform) and Duplex. A Through Stream is a sink stream that reads what goes into the Source Stream, it can also be written to. A duplex stream is a pair of streams ({Source, Sink}) streams.

Pull-Streams

Source Streams

A Source Stream (aka readable stream) is a async function that may be called repeatedly until it returns a terminal state. Pull-streams have back pressure, but it implicit instead of sending an explicit back pressure signal. If a source needs the sink to slow down, it may delay returning a read. If a sink needs the source to slow down, it just waits until it reads the source again.

	var request = require('request')
	request.get('http://api.twitter.com/1/users/show.json?screen_name=mikeal', {json:true}, function (e, r, doc) {
	console.log(doc)
	})
	request.get('https://api.twitter.com/1/statuses/user_timeline.json?include_entities=true&include_rts=true&screen_name=mikeal', {json:true}, function (e, r, doc) {
	console.log(doc)
	})

	// data comes from here http://stat-computing.org/dataexpo/2009/the-data.html
	// download 1994.csv.bz2 and unpack by running: cat 1994.csv.bz2 \| bzip2 -d > 1994.csv
	// 1994.csv should be ~5.2 million lines and 500MB
	// importing all rows into leveldb took ~50 seconds on my machine

	// there are two main techniques at work here:
	// 1: never create JS objects, leave the data as binary the entire time (binary-split does this)
	// 2: group lines into 16 MB batches, to take advantage of leveldbs batch API (byte-stream does this)

	var level = require('level')