specs for a mongodb query -> process balancer invoking workers

load.md

balancer code

• load a config

{
  "collection" : "products",
  "match" : { "language" : "de" },
  "queueProperty" : "__mdbqb",
  "cleanUpQueueProperty" : false,
  "allocation" : {
    "command" : "sh command [$__mdbqb.worker]",
    "inParallel" : 10
  }
}

__mdbqb specs

this will be appended to a item inside the collection

 item."queueProperty[__mdbqb]"  = {
  "worker" : "p1",
  "status" : "inprogress|done|open|error",
  "started" : new Date(),
  "finisehd" : new Date()
}

• connect to db
• query by "match" and splice results into "inParallel[10]" pieces
• iterate pieces and update items in pieces, set "queueProperty[__mdbqb.worker]" = "p" + [i] to add it to a queue
• loop "inParallel[10]": start invocation "command" sh command [$__mdbqb.worker]"
• use https://gist.github.com/vladignatyev/06860ec2040cb497f0f3 to show progress
• get progress infos from subscripts by https://stackoverflow.com/questions/4760215/running-shell-command-and-capturing-the-output
• if all workers are ready and "cleanUpQueueProperty" === true : $unset item."queueProperty[__mdbqb]"

worker code

to be compatible a worker needs to stick to an API spec

• receive "queueProperty[__mdbqb]" "worker"
• query for it and update [$__mdbqb.worker]" = p[i] and [$__mdbqb.status]" = "inprogress|done|open|error" and [$__mdbqb.started]" = new Date and [$__mdbqb.started]" = new Date
• stdout iterator for current item queried by [$__mdbqb.worker]