Simple helper to use a pool of worker asynchronous functions to handle of queue of tasks. Each done worker will be re-used to handle the next task.

pooltask.md

pooltask

Simple helper to use a pool of worker asynchronous functions to handle of queue of tasks. Each done worker will be re-used to handle the next task.

Callback use the node.js format:

  cb = function(err, result...){...}

Example with dummy asynchronous functions:

var push = pooltask(2, function(param, cb)
{
    setTimeout(function(){
        console.log("Task %s done",param);
        cb('job done!');
    },1);
}, function(err){
    console.log("Done");
});

push(123);
push(234);
push(345);

pooltask

function push = pooltask( numeric free, function worker, function done )

Create the pooltask helper an return a function. Call it to push a new task in the queue.

• numeric *free:

  The number of parallel worker.

• function worker:

  The asynchronous worker function. See below.

• function done:

  Will be called every times the number of free worker reach the free value.

  It can happens many times, it depends on the way you using the push function.

worker

function( ..., function next ):

The worker function. It will be called every time the current number of worker didn't reach the free value AND when there is some task in the queue.

• ...:

  An arbitrary list of arguments passed at the push function. If 2 arguments are passed to push, two arguments will be available in the worker function.

• function next:

  The callback to continue with the next task OR call done.

next

function(? err)

The callback passed at last argument for the worker used to continue the process.

• ? err:

  To indiquate an error, a asynchronous function should call next with something for it's first argument err. It will call done by passing this err argument. See below.

  To indiquate a success, pass a null value for err argument.

  Any type can be used but remember: any different value from null will be considered as an error.

done

function(? err)

The final callback, called when the last worker call its next callback AND when there is no more task in the queue. In that case the done function will be called.

Note: this function can be called many times. It depends how the push function is called.

• ? err:

  When any worker asynchronous function send an error.

push

function(...):

The initial function that push task in the queue.

• ...:

pooltask.js

/** pooltask
 *
 * Simple helper to use a pool of worker asynchronous functions to handle of queue of tasks.

 * Each done worker will be re-used to handle the next task.
 * Callback use the node.js format.
 * Syntax:
 * - function *init* = waterfall(array *funcs*, function *done*)
 * - array *funcs* = [ function(..., function *next*), ... ]
 * - function *next* = function(*err*, ...)
 * - function *done* = function(*err*, ...)
 * - function *init* = function(*err*, ...)
 */
var pooltask = function(free, worker, cb)
{
    var slice = Array.prototype.slice;
    var full = free;
    var pool = [];
    var error = null;
    var done = function(err)
    {
        free += 1;
        next.apply(null, arguments);
    }
    var next = function(err)
    {
        if(err || error) return error || ((error = err) && cb && cb(err));
        if(free == full && pool.length == 0) return cb(null);
        if(free < 1 || !(args = pool.pop())) return;
        free -= 1;
        args.push(done);
        worker.apply(null, args);
    };
    var push = function()
    {
        pool.push(slice.call(arguments));
        next();
    };
    return push;
};