Skip to content

Instantly share code, notes, and snippets.

@sengstacken

sengstacken/read data from s3

Created September 25, 2020 01:37
Show Gist options
  • Save sengstacken/9c7021c6ddfcdc324b7bc3adfcf7c08d to your computer and use it in GitHub Desktop.
Save sengstacken/9c7021c6ddfcdc324b7bc3adfcf7c08d to your computer and use it in GitHub Desktop.
Download ZIP
read data from s3
Raw
read data from s3
#loop over all docs in the bucket
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
for obj in bucket.objects.all():
if obj.key.split('/')[0].split('.')[-1] == 'txt':
obj_from_s3 = s3.Object(bucket, key)
file_content = obj_from_s3.get()['Body'].read().decode('utf-8')
json_content = json.loads(file_content)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment