Skip to content

Instantly share code, notes, and snippets.

@inc0
Created April 23, 2020 21:55
Show Gist options
  • Save inc0/53617c59bb977c3d9e1ba634c8317602 to your computer and use it in GitHub Desktop.
Save inc0/53617c59bb977c3d9e1ba634c8317602 to your computer and use it in GitHub Desktop.
defaults = {
"image": "argopy:latest"
}
@argopy.container(**defaults)
def process_data(data_partition):
do something with data
partitions = ['foo.csv', 'bar.csv']
with argopy.run as ap:
for part in partitions:
process_data(part)
ap.run()
# Example of loops using DAGs
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: loops-dag-
spec:
entrypoint: loops-dag
templates:
- name: loops-dag
dag:
tasks:
- name: process_data
template: process_data
with_items:
- foo.csv
- bar.csv
- name: process_data
inputs:
parameters:
- name: partition
container:
image: argopy:latest
command: [argopy-entrypoint]
args: ["{{inputs.parameters.partition}}"]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment