Wrap boto3 dynamodb in an exponential backoff to avoid ProisionedThroughputExceededException

dynamodb_read_backoff.py

#!/usr/bin/env python

# An exponential backoff around Boto3 DynamoDB, whose own backoff eventually
# fails on long multipage scans. We'd like to use this as a wrapper somehow,
# see: https://gist.github.com/numberoverzero/cec21b8ca715401c5662

from time import sleep

import boto3
from boto3.dynamodb.conditions import Attr
from botocore.exceptions import ClientError

RETRY_EXCEPTIONS = ('ProvisionedThroughputExceededException',
                    'ThrottlingException')

dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('cshenton-exception-test')

# from datetime import datetime
# for i in range(100000):
#     x = table.put_item(Item={'id': str(i), 'dt': datetime.now().isoformat()})
#     if not i % 100:
#         print(i)

# We cannot begin with ExclusiveStartKey=None, so we use kwargs sans that the
# first time, then update to include it subsequently.
scan_kw = {'FilterExpression': Attr('dt').contains(':00')}  # no StartKey yet
retries = 0
popular = []
while True:
    try:
        res = table.scan(**scan_kw)
        popular.extend(res['Items'])
        last_key = res.get('LastEvaluatedKey')
        print('len={} res[Count]={} last_key={}'.format(
            len(popular), res['Count'], last_key))
        if not last_key:
            break
        retries = 0          # if successful, reset count
        scan_kw.update({'ExclusiveStartKey': last_key})
    except ClientError as err:
        if err.response['Error']['Code'] not in RETRY_EXCEPTIONS:
            raise
        print('WHOA, too fast, slow it down retries={}'.format(retries))
        sleep(2 ** retries)
        retries += 1     # TODO max limit

I think botocore already implements an exponential backoff, so when it gives the error it already did its max tries (max tries can be configured)

I think botocore already implements an exponential backoff, so when it gives the error it already did its max tries (max tries can be configured)

Yes it was my understanding that ALL AWS SDK's have to implement exponential backoff

The SDKs do implement exponential backoff, however we have found sometimes we need more time than the default number of retries gives us.

Turns out there's an under-documented feature you can use to config the number of retries (as mojimi hints), which exponentially increase the backoff time. We've used this for DynamoDB as well as other services (e.g., Textract OCR). I wish I'd found this earlier: it makes my wrapper code unnecessary.

Here's a StackOverflow that shows the pattern (for 'ec2'), with pasted code:
https://stackoverflow.com/questions/34003350/change-the-number-of-request-retries-in-boto3#48568320

from botocore.config import Config
config = Config(
    retries = dict(
        max_attempts = 10
    )
)
ec2 = boto3.client('ec2', config=config)

Thanks, that is very useful to know!