Skip to content

Instantly share code, notes, and snippets.

@ngollperrier

ngollperrier/s3-partition-discovery.py

Created April 23, 2021 18:00
Show Gist options
  • Save ngollperrier/a38a160057415d70e65cfc21b462d098 to your computer and use it in GitHub Desktop.
Save ngollperrier/a38a160057415d70e65cfc21b462d098 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
s3-partition-discovery.py
# Acceptable events for processing (other are discarded
TRIGGERING_EVENTS = ['ObjectCreated:Put', 'ObjectCreated:CompleteMultipartUpload']
# The S3 path inference regexp: env/topics/domain/raw/format/table/partition_scheme
EXTRACTION_REGEXP = '(?P<env>.*)/topics/(?P<domain>.*)/raw/(?P<format>.*)/(?P<short_name>.*)/' \
'(?P<version_partition>.*)=(?P<version>.*)/' \
'event_date=(?P<event_date>.*)/event_hour=(?P<event_hour>.*)/' \
'(?P<file>.*)'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment