Skip to content

Instantly share code, notes, and snippets.

@evanderkoogh
Created November 20, 2017 13:29
Show Gist options
  • Save evanderkoogh/37f6adb01268e82ecd68d46fce1c7c5a to your computer and use it in GitHub Desktop.
Save evanderkoogh/37f6adb01268e82ecd68d46fce1c7c5a to your computer and use it in GitHub Desktop.
Import CSV into DynamoDB
whateverId attribute1 someotherattribute
foo bar baz
hello erwin world
const fs = require('fs')
const parse = require('csv-parse/lib/sync')
const AWS = require('aws-sdk')
AWS.config.update({region: 'ap-southeast-2'});
const docClient = new AWS.DynamoDB.DocumentClient()
const contents = fs.readFileSync('./<filename>.csv', 'utf-8')
// If you made an export of a DynamoDB table you need to remove (S) etc from header
const data = parse(contents, {columns: true})
data.forEach((item) => {
if(!item.maybeempty) delete item.maybeempty //need to remove empty items
docClient.put({TableName: '<Table>', Item: item}, (err, res) => {
if(err) console.log(err)
})
})
{
"name": "dydbimport",
"version": "1.0.0",
"main": "index.js",
"license": "MIT",
"dependencies": {
"aws-sdk": "^2.153.0",
"csv-parse": "^2.0.0"
}
}
@rubenberna
Copy link

Hi,

When processing the file, I get an error for 'Type mismatch':

'One or more parameter values were invalid: Type mismatch for key externalId expected: N actual: S'

How do you define the data type in the CSV or overcome this error altogether?

Thanks

@shlomizadok
Copy link

@rubenberna - I have had an integer column which was part of an index.
I have modified the above to

data.forEach((item) => {
  if(!item.maybeempty) delete item.maybeempty //need to remove empty items
  item.added = parseInt(item.added); // <-- Changed this to Integer solved the Type mismatch error
  docClient.put({TableName: 'ReviewTable', Item: item}, (err, res) => {
    if(err) console.log(err)
  })      
})

@BartusZak
Copy link

BartusZak commented Sep 7, 2020

node_modules\csv-parse\lib\sync.js:30
    throw err;
    ^

Error: Invalid opening quote at line 2

@bradbyte
Copy link

This should now be ...

const {parse} = require('csv-parse/sync');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment