osm-p2p-syncfile integration

integration.md

syncfile integration

The new syncfile module is called osm-p2p-syncfile. It manages a specially formatted tarball that stores media blobs as well as a P2P database (currently osm-p2p-db).

basic usage

Desktop and mobile will both need to open a syncfile (new or existing) and sync the locally running database with it. This'll probably look something like

var Syncfile = require('osm-p2p-syncfile')
var blobSync = require('blob-store-replication-stream')
var tmpdir = require('os').tmpdir()  // where the p2p db gets temporarily extracted to (media stays put in the syncfile)

App.prototype.syncWithFile = function (syncfilePath, cb) {
  var self = this
  var syncfile = new Syncfile(syncfilePath, tmpdir)
  var pending = 2
  var error

  syncfile.ready(function () {
    // sync the p2p db
    replicate(self.osm.log.replicate({live: false}),
              syncfile.osm.log.replicate({live: false}),
              done)

    // sync the media
    replicate(blobSync(self.media),
              blobSync(syncfile.media),
              done)
  })
  
  function done (err) {
    if (err) error = err
    if (!--pending) {
      syncfile.close(function () {
        cb(error)
      })
    }
  }
})

function replicate (stream1, stream2, cb) {
  stream1.on('end', done)
  stream1.on('error', done)
  stream2.on('end', done)
  stream2.on('error', done)

  stream1.pipe(stream2).pipe(stream1)

  var pending = 2
  var error
  function done (err) {
    error = err || error
    if (!--pending) cb(err)
  }
}

database type

You can use the syncfile.userdata API to set arbitrary fields on the syncfile. We could use this to record the p2p db type when we create a syncfile, like

var syncfile = new Syncfile(syncfilePath, tmpdir)
syncfile.ready(function () {
  syncfile.userdata({p2pDbType: 'osm-p2p-db'}, function () {
    syncfile.close()
  })
})

desktop

The desktop app needs to handle old-style syncfiles as well as the new osm-p2p-syncfile format.

The former stores itself as a gzipped tarball, and the latter as a regular tarball. Tar archives don't have a magic number, but gzip does! A gzip archive begins with the two bytes 1f 8b. The desktop client could use this to determine if the user has selected an old syncfile or new. There are some modules for this, I bet.

@Karissa would mapeo-sync be the integration point for this?

@noffle yeah i think so, although I'm leaning towards the idea that mapeo-sync is an improper abstraction. What would be ideal is a mapeo-core library that is mostly just the functions from mapeo-server abstracted out for use beyond a node server. mapeo-sync was an early attempt at this but then we realized we needed to add a lot more features than just sync