Skip to content

Instantly share code, notes, and snippets.

View rgpower's full-sized avatar

Rick Power rgpower

5 followers · 0 following
  • Halifax, NS
View GitHub Profile
@rgpower
rgpower / s3_lambda_event_template.md
Last active October 17, 2020 12:14
Using an event template json file, generate a bunch lambda trigger events ,to feed into a lambda, for processing using docker via lambci/lambda

you have a list of s3 objects in a bucket that need to be imported manually into BigQuery/BQ for backfill etc, named missing.txt that contains items like

exports/project-activity/firehose/2020/09/28/12/project-activity-batch-s3-1-2020-09-28-12-58-57-56b1c279.gz
exports/project-activity/firehose/2020/09/28/13/project-activity-batch-s3-1-2020-09-28-13-14-00-9e633629.gz
exports/project-activity/firehose/2020/09/28/13/project-activity-batch-s3-1-2020-09-28-13-29-01-f3174b15.gz

take a template file like this, say, named event-json.tpl,

{
  "Records": [