DynamoDB Scan paging w/ Ruby SDK

dynamo_scan_paging_with_ruby_sdk.md

Resources

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Scan.html

http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB/Client.html

Summary

DynamoDB's scan operation fetches only 1MB at one time and it adds LastEvaluatedKey (LEK) in the response if it has a next page.

I didn't know the spec and troubled a bit in my project.

Solution

items = []

scan_opts = {
  table_name: "MyDynamoTable",
  scan_filter: {
    "PublishedAt" => {
      attribute_value_list: ["2017-08-25"],
      comparison_operator: "GE",
    },
    "Host" => {
      attribute_value_list: ["example.com"],
      comparison_operator: "EQ",
    }
  },
  consistent_read: true,
}

scan_output = client.scan scan_opts

loop do
  items << scan_output.items

  break unless (lek = scan_output.last_evaluated_key)

  scan_output = client.scan scan_opts.merge(exclusive_start_key: lek)
end

items.flatten

@michael-bowen-sp thanks for comment!

This is an old code and I have no re-pro environment now. but you mean I should do like the following?

loop do
  items << scan_output.items

  break unless (lek = scan_output.last_evaluated_key)

  scan_output = client.scan scan_opts.merge(exclusive_start_key: lek)
end

If I'm wrong, please tell me in code 😄

@mmyoji it looks like a working solution. I suggest to update your gist with the way you mentioned in comment.

thanks a lot. for newbies it helps a lot.

@miry I updated original one. Thanks 👍

@mmyoji If you follow the method of passing in the page and the number of items on a page, how should you write it? @@ Thanksss