Skip to content

Instantly share code, notes, and snippets.

@Geremie
Created April 4, 2022 00:02
Show Gist options
  • Save Geremie/fcea3aa4a5c56b35cef51c9a852847f9 to your computer and use it in GitHub Desktop.
Save Geremie/fcea3aa4a5c56b35cef51c9a852847f9 to your computer and use it in GitHub Desktop.
Are you using Cloud Functions for event based processing ?
def files_processor():
wait_for_objects_task = MultipleGcsBucketSensor(
task_id='wait_for_objects',
buckets=['sandbox-mdjohossou-first-bucket', 'sandbox-mdjohossou-second-bucket']
)
process_objects_task = PythonOperator(
task_id='process_objects',
python_callable=process_objects,
provide_context=True
)
trigger_files_processor_dag_task = TriggerDagRunOperator(
task_id='trigger_files_processor_dag',
trigger_dag_id='files_processor'
)
wait_for_objects_task >> process_objects_task >> trigger_files_processor_dag_task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment