Import CSV into DynamoDB

example.csv

index.js

const fs = require('fs')
const parse = require('csv-parse/lib/sync')
const AWS = require('aws-sdk')

AWS.config.update({region: 'ap-southeast-2'});
const docClient = new AWS.DynamoDB.DocumentClient()

const contents = fs.readFileSync('./<filename>.csv', 'utf-8')
// If you made an export of a DynamoDB table you need to remove (S) etc from header
const data = parse(contents, {columns: true})

data.forEach((item) => {
        if(!item.maybeempty) delete item.maybeempty //need to remove empty items
        docClient.put({TableName: '<Table>', Item: item}, (err, res) => {
                if(err) console.log(err)
        })      
})

package.json

{
  "name": "dydbimport",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "dependencies": {
    "aws-sdk": "^2.153.0",
    "csv-parse": "^2.0.0"
  }
}

What is csv-parse/lib/sync

@SaiiKiran, csv-parse is an NPM package.

It can be installed into a project with:

npm i -S csv-parse

Here's the package page: https://www.npmjs.com/package/csv-parse.

Add those lines if you need to set a profile using ~/.aws/config:

var credentials = new AWS.SharedIniFileCredentials({profile: 'profile-name'});
AWS.config.credentials = credentials;

Hi,
I received the following error, I think there are limits in memory.
I imagine this line:
const data = parse(contents, {columns: true})
want to work in RAM all rows.
My file is 450MB
what to you think to put record lines by lines?

<--- Last few GCs --->

[25097:0x35e00f0] 114840 ms: Mark-sweep 1632.6 (1687.3) -> 1631.7 (1688.8) MB, 420.6 / 0.0 ms allocation failure GC in old space requested
[25097:0x35e00f0] 115253 ms: Mark-sweep 1631.7 (1688.8) -> 1631.6 (1658.3) MB, 412.9 / 0.0 ms last resort GC in old space requested
[25097:0x35e00f0] 115651 ms: Mark-sweep 1631.6 (1658.3) -> 1631.6 (1658.3) MB, 398.0 / 0.0 ms last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x1cd27f2a58b9
1: __onRow [./node_modules/csv-parse/lib/index.js:~560] [pc=0x2f17ce990a37](this=0xcabe84da141 )
2: __parse [./node_modules/csv-parse/lib/index.js:~348] [pc=0x2f17ce98e711](this=0xcabe84da141 ,nextBuf=0xcabe84da209 ,end=0x1447c5402...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [node]
2: 0x8ccf9c [node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
5: v8::internal::Factory::NewUninitializedFixedArray(int) [node]
6: 0xd81903 [node]
7: v8::internal::Runtime_GrowArrayElements(int, v8::internal::Object**, v8::internal::Isolate*) [node]
8: 0x2f17ce8042fd

Hi,

When processing the file, I get an error for 'Type mismatch':

'One or more parameter values were invalid: Type mismatch for key externalId expected: N actual: S'

How do you define the data type in the CSV or overcome this error altogether?

Thanks

@rubenberna - I have had an integer column which was part of an index.
I have modified the above to

data.forEach((item) => {
  if(!item.maybeempty) delete item.maybeempty //need to remove empty items
  item.added = parseInt(item.added); // <-- Changed this to Integer solved the Type mismatch error
  docClient.put({TableName: 'ReviewTable', Item: item}, (err, res) => {
    if(err) console.log(err)
  })      
})

node_modules\csv-parse\lib\sync.js:30
    throw err;
    ^

Error: Invalid opening quote at line 2

This should now be ...

const {parse} = require('csv-parse/sync');