PhillyCDO

See government announcement [here] (http://www.ci.south-bend.in.us/residents-business-government/news/2013-8-21/mayor-buttigieg-launch-city%E2%80%99s-open-data-portal-website).

###CITY OF SOUTH BEND, INDIANA

###EXECUTIVE ORDER NO. 2-2013

###ESTABLISHMENT OF OPEN DATA PORTAL FOR PUBLIC INFORMATION

WHEREAS, this Administration is committed to timely and consistent publication of data consistent with the Indiana Access to Public Records Act (APRA) which is essential to open and effective government; and

PhillyCDO

// data comes from here http://stat-computing.org/dataexpo/2009/the-data.html
// download 1994.csv.bz2 and unpack by running: cat 1994.csv.bz2 | bzip2 -d > 1994.csv
// 1994.csv should be ~5.2 million lines and 500MB
// importing all rows into leveldb took ~50 seconds on my machine

// there are two main techniques at work here:
// 1: never create JS objects, leave the data as binary the entire time (binary-split does this)
// 2: group lines into 16 MB batches, to take advantage of leveldbs batch API (byte-stream does this)

var level = require('level')