Check if a payload you plan to store falls under the 400KB DynamoDB item size limit

DynamoDB Item Size Limit Checker

A quick utility for checking whether you should store an item in DynamoDB.

DynamoDB currently limits each item stored to a maximum of 400KB.

Continue reading to explore how to check if a payload you plan to store falls under the 400KB DynamoDB item size limit.

There are two approaches for this:

Ship it anyway and let it fail, then catch the exception.
Or, try to approximate the item size yourself, and do other logic instead if you need to.

Both approaches are covered here.

This has been tested with the AWS JavaScript SDK (V3) as of January 12, 2021.

First approach: shipping it anyway and catching the exception:

try {
  // just do dynamoDbClient.send(PutItemCommmand) or .send(UpdateItemCommand) or whatever...
} catch (error) {
  if (error instanceof Error &&  error.name === 'ValidationException' &&
       error.message === 'Item size has exceeded the maximum allowed size') {
    return {
      code: 400,
      message: 'Bad Request. That is a huge payload and it will not fit as one item entry in DynamoDB!'
    }
  }
  throw error
}

Second approach: approximating the item size:

Quick usage example:

try {
  // define some item like:
  const item = throwIfDynamoDbPayloadIsTooLarge({
    id: 'abc123',
    data: 'xyz123'
  })
  // if you don't want to throw, you can also use `isValidDynamoDbPayloadSize(...)`
  // then just do dynamoDbClient.send(PutItemCommmand) or .send(UpdateItemCommand) or whatever...  
} catch (error) {
  if (error instanceof Error && error.name === 'DynamoDbPayloadIsTooLargeError') {
    return {
      code: 400,
      message: 'Bad Request. That is a huge payload and it will not fit as one item entry in DynamoDB!'
    }
  }
  // Obviously, you should still also handle the ValidationException,
  // like in the first example, if you end up making the request.
  throw error
}

And the supporting functions for the usage example above:

import { strictEqual } from 'assert'

// Testing the functions below with the example from:
// https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ServiceQuotas.html#limits-items
const itemForSizeExample = {
  'shirt-color': 'R',
  'shirt-size': 'M'
}

strictEqual(getApproximateDynamoDbItemSizeInBytes(itemForSizeExample), 23, 'It looks like the DynamoDB Item Size approximator is broken!')
strictEqual(isValidDynamoDbPayloadSize(itemForSizeExample), true, 'It looks like the DynamoDB Item Size approximator is broken!')

/** Returns string length in bytes */
export function getBytes (s: string): number {
  return Buffer.byteLength(s, 'utf8')
}

/** Returns object length by calculating the size of the attribute names and values, similar to how DynamoDB does it */
export function getApproximateDynamoDbItemSizeInBytes (item: object): number {
  return Object.entries(item).reduce((size, entry) => {
    return size + getBytes(entry[0]) + getBytes(JSON.stringify(entry[1]))
  }, 0)
}

/** Checks if the provided payload is under 400KB (400 * 1000 = 400000 bytes) */
export function isValidDynamoDbPayloadSize (payload: object) {
  return (typeof payload === 'object' && getApproximateDynamoDbItemSizeInBytes(payload) < (400 * 1000)) ? true : false
}

/** An exception indicating the payload is too large for DynamoDB */
export class DynamoDbPayloadIsTooLargeError extends Error {
  constructor () {
    super('The data in the request is too large.')
    this.name = 'DynamoDbPayloadIsTooLargeError'
  }
}

/** A helper function that automatically checks the provided payload for size, then passes it through or errors */
export function throwIfDynamoDbPayloadIsTooLarge <Payload extends object> (payload: Payload): Payload {
  if (!isValidDynamoDbPayloadSize(payload)) throw new DynamoDbPayloadIsTooLargeError()
  return payload
}

It is worth noting that the approximation method seems to error out a few hundred bytes before the actual limit. This is because documentation is lacking on the exact calculation to perform to get the same calculation as AWS. This approximation is good enough though and errors out just early enough before you hit the AWS limit.

Read more about the way item size is measured by AWS for DynamoDB: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ServiceQuotas.html#limits-items

khalidx/dynamodb-item-size-limit.md

DynamoDB Item Size Limit Checker