Multiple outputs workaround luigi.

README.md

README

Make sure luigi is importable.

Run:

$ python main.py ProcessFiles --local-scheduler

After Input, Unzipper and ProcessFiles completed successfully, there should find a list of file sizes:

$ cat zip.filelist.sizes.txt
./tmp/GMT-0: 62
./tmp/Eire: 1328
./tmp/GMT+0: 62
...

Clean:

$ make clean

main.py

#!/usr/bin/env python
# coding: utf-8

import zipfile
import luigi
import os

class Input(luigi.ExternalTask):
    """
    A zipfile.
    """

    def output(self):
        return luigi.LocalTarget("zoneinfo.zip")

class Unzipper(luigi.Task):
    """
    Unzips a file and creates a single file, that contains the extracted paths. 
    """

    def requires(self):
        return Input()

    def run(self):
        """
        Extract zipped files into some temporary dir.
        """
        zfile = zipfile.ZipFile(self.input().path)
        for name in zfile.namelist():
            zfile.extract(name, './tmp')
        zfile.close()

        # create a list of extracted paths and write them to output
        with self.output().open('w') as output:
            for root, dirs, files in os.walk('./tmp'):
                for fn in [os.path.join(root, name) for name in files]:
                    output.write("%s\n" % fn)

    def output(self):
        return luigi.LocalTarget("zip.filelist.txt")

class ProcessFiles(luigi.Task):
    """
    Do something with each extracted file.
    """

    def requires(self):
        return Unzipper()

    def run(self):
        """ Just dump the file size of each file. """
        with self.input().open() as handle:
            with self.output().open('w') as output:
                for line in handle:
                    filename = line.strip()
                    output.write("%s: %s\n" % (filename, os.stat(filename).st_size))

    def output(self):
        return luigi.LocalTarget("zip.filelist.sizes.txt")

if __name__ == '__main__':
    luigi.run()

Makefile

all:
	python main.py ProcessFiles --local-scheduler

clean:
	rm -rf tmp/ zip.filelist.*

zoneinfo.zip