mikeal

SLEEP

A proposed simple specification for syncing tabular data over HTTP via JSON transaction logs.

(Originally drafted explicitly by Christopher Groskopf of the Panda Project and Francis Irving of ScraperWiki on 21st December 2011. Concept by Max Ogden. Named by Isaac Z. Schlueter)

Motivation document by Max Ogden: https://gist.github.com/1246804

mikeal

Apache CouchDB's data model stores a "revision tree". This tree branches on two events:

• the same revision is updated on two nodes and then those node's replicate, creating a conflict.
• a document is deleted by writing a new revision with _deleted, then the document is created again which creates a new tree begining with a revision starting in 1-.

Because CouchDB also implements a default "most writes wins" policy as a default winner during conflicts and replication this means that during replication the client must read the entire rev tree up to the point that the last matching revision exists.

In practice, very few CouchDB applications actually resolve conflicts using the revision. Instead, the vast majority of applications just stick with the "most writes wins" policy which works well enough for most workloads.

So, for couchup, I've decided to rely on the "most writes wins" policy and not write or hold on to the entire revision tree. Since I'm not storing the entire rev tree I do need an alternative

mikeal

/getable/public/2dfd9a2c08e6f83ad6c50aeb932f927c null

/Users/mikeal/Documents/git/constructable/scripts/manta.js:52
              if (err) throw err
                             ^
Error: socket hang up
    at createHangUpError (http.js:1417:15)
    at CleartextStream.socketCloseListener (http.js:1467:23)
    at CleartextStream.EventEmitter.emit (events.js:117:20)
    at tls.js:606:10

mikeal

6/23/2013

Last weekend I spent a few hours implementing the CouchDB document storage model on top of levelup in node.js.

Today I had a few spare hours in my Sunday and decided to implement the HTTP interface and compare the performance with Apach CouchDB. The benchmarks can be found here https://github.com/mikeal/couchup/tree/master/tests/benchmark

node compare-new-writes.js

Apache CouchDB 67140 writes

mikeal

(master)⚡ % npm install --production sendgrid                                                                                                                      ~/Documents/git/constructable
npm WARN package.json [email protected] No README.md file found!
npm http GET https://registry.npmjs.org/sendgrid
npm http 304 https://registry.npmjs.org/sendgrid
npm http GET https://registry.npmjs.org/step/0.0.5
npm http GET https://registry.npmjs.org/chai/1.5.0
npm http GET https://registry.npmjs.org/nock/0.16.0
npm http GET https://registry.npmjs.org/sinon/1.6.0
npm http GET https://registry.npmjs.org/mocha/1.9.0
npm http GET https://registry.npmjs.org/underscore/1.4.4

mikeal

function MyObj () {
}
MyObj.prototype.test = function (one, two, three) { return three}

function bench () {
  var x = new MyObj()
  var y = x.test.bind(x)
  y(1, 2, 3)
}

mikeal

https://gist.github.com/mikeal/5586193
https://gist.github.com/mikeal/5588540

mikeal

function MyObject (val) {this.val = val}

function createMyObject (val) {return new MyObject(val)}

function bench () {
  createMyObject(1)
}

mikeal

var j = JSONStream.parse('rows.*.doc')
j._write = j.write
response.pause = function () {}
j.write = function (chunk) {j._write(chunk); return true;}
response.on('data', j.write.bind(j))
// response.pipe(j)
j.on('data',  console.log.bind(console))
response.on('end', console.log.bind(console, 'end'))

mikeal

var fs = require('fs')
  , path = require('path')
  , http = require('http')
  , browserify = require('browserify')
  , request = require('request')
  ;

http.createServer(function (req, resp) {
  resp.statusCode = 200
  if (req.url === '/test.js') {