Boto3 S3 constant batch example

constant_batch.py

import boto3

BATCH_SIZE = 1024

s3 = boto3.resource('s3')
s3_object = s3.Object('test-bucket', 'example_object')

stream = s3_object.get()['Body']

batch = stream.read(BATCH_SIZE)
while batch:
  result = do_some_work(batch)
  batch = stream.read(BATCH_SIZE)

Note this code is not tested, and contains a free reference to the function do_some_work.

The problem with this approach is that you have to be able to deal with fixed sized chunks, so you'll need some buffering in there to find actual record boundaries.