Skip to content

Instantly share code, notes, and snippets.

@JeremyPlease
Last active April 9, 2019 22:20
Show Gist options
  • Save JeremyPlease/db55ad0f0336df4aa9d9c8a57f74688f to your computer and use it in GitHub Desktop.
Save JeremyPlease/db55ad0f0336df4aa9d9c8a57f74688f to your computer and use it in GitHub Desktop.
Redact phone and email from large json file
const fs = require('fs')
const emailRegex = /@[A-z0-9\.-]{3,50}/g
const phoneRegex = /\+[\d\s-]{4}/g
const liveRegex = /live/g
const file = '/path/to/big/json/file.json'
const outputFile = '/path/to/output/redacted.json'
const inputStream = fs.createReadStream(file, {encoding: 'utf8'})
const outputStream = fs.createWriteStream(outputFile, {encoding: 'utf8'})
inputStream.on('data', (data) => {
outputStream.write(
data.replace(emailRegex, '@example.com')
.replace(phoneRegex, '+1555')
.replace(liveRegex, 'staging')
)
})
inputStream.on('end', () => process.exit(0))
fs.watchFile(outputFile, () => {
fs.stat(outputFile, (err, stats) => {
console.log((stats.size/1000000000) + 'GB')
})
})
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment