- load a config
{
"collection" : "products",
"match" : { "language" : "de" },
"queueProperty" : "__mdbqb",
"cleanUpQueueProperty" : false,
"allocation" : {
"command" : "sh command [$__mdbqb.worker]",
"inParallel" : 10
}
}
this will be appended to a item inside the collection
item."queueProperty[__mdbqb]" = {
"worker" : "p1",
"status" : "inprogress|done|open|error",
"started" : new Date(),
"finisehd" : new Date()
}
- connect to db
- query by "match" and splice results into "inParallel[10]" pieces
- iterate pieces and update items in pieces, set "queueProperty[__mdbqb.worker]" = "p" + [i] to add it to a queue
- loop "inParallel[10]": start invocation "command" sh command [$__mdbqb.worker]"
- use https://gist.github.com/vladignatyev/06860ec2040cb497f0f3 to show progress
- get progress infos from subscripts by https://stackoverflow.com/questions/4760215/running-shell-command-and-capturing-the-output
- if all workers are ready and "cleanUpQueueProperty" === true : $unset item."queueProperty[__mdbqb]"
to be compatible a worker needs to stick to an API spec
- receive "queueProperty[__mdbqb]" "worker"
- query for it and update [$__mdbqb.worker]" = p[i] and [$__mdbqb.status]" = "inprogress|done|open|error" and [$__mdbqb.started]" = new Date and [$__mdbqb.started]" = new Date
- stdout iterator for current item queried by [$__mdbqb.worker]