Skip to content

Instantly share code, notes, and snippets.

@eugenehp
Created January 22, 2018 06:28
Show Gist options
  • Save eugenehp/d23e542ce091958827ff2f88e19b4fc4 to your computer and use it in GitHub Desktop.
Save eugenehp/d23e542ce091958827ff2f88e19b4fc4 to your computer and use it in GitHub Desktop.
How to insert million records via nodejs at maximum speed via pipe
const knex = require('knex')(knexConfig)
const csv = require('csv-parser')
const Transform = require('stream').Transform;
var fs = require('fs');
function importData(knex, Promise) {
return new Promise(function (resolve, reject){
fs.createReadStream('some-csv-file.csv')
.on('error', reject)
.pipe(csv())
.on('error', reject)
.pipe(new Transform({
objectMode: true,
transform: function (chunk, _, next) {
knex(tableName).insert(chunk).then(function () {
next();
}, next);
}, function (done) {
resolve();
done();
})).on('error', reject);
}));
});
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment