Skip to content

Instantly share code, notes, and snippets.

@samuell
Last active August 29, 2015 14:04
Show Gist options
  • Select an option

  • Save samuell/476867703a3ee7ab4c3a to your computer and use it in GitHub Desktop.

Select an option

Save samuell/476867703a3ee7ab4c3a to your computer and use it in GitHub Desktop.
class TaskA(luigi.Task):
def output(self):
# Multiple, named output TARGETS
return { 'out1' : luigi.LocalTarget('out1.txt'),
'out2' : luigi.LocalTarget('out2.txt') }
...
class TaskB(luigi.Task):
def output(self):
# Multiple, named output TARGETS
return { 'out1' : luigi.LocalTarget('out1.txt'),
'out2' : luigi.LocalTarget('out2.txt') }
...
class TaskC(luigi.Task):
def requires(self):
# Multiple, named input TASKS
return { 'in1' : TaskA(),
'in2' : TaskB() }
def run(self):
# Note especially how we have to look up a 2-step nested dict
# structure (The upstream TASK, and the specific TARGET of that
# task, that we want) when using the input target:
with open(self.input()['in1']['out2'].path) as infile:
for line in infile:
do_something(line)
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment