Skip to content

Instantly share code, notes, and snippets.

@Khazl
Created November 29, 2020 11:49
Show Gist options
  • Save Khazl/b00e392c5f72639270d771e68e60c48e to your computer and use it in GitHub Desktop.
Save Khazl/b00e392c5f72639270d771e68e60c48e to your computer and use it in GitHub Desktop.
/*
Usage:
$ node crawler.js <url> <filename>
Example:
$ node crawler.js "https://jsonplaceholder.typicode.com/users" "users.json"
*/
const https = require('https');
const fs = require('fs');
const url = process.argv[2];
const filename = process.argv[3];
console.log("\x1b[32mRequest to: \x1b[37m" + url);
console.log("\x1b[32mSave to: \x1b[37m" + filename);
https.get(url, (res) => {
let body = '';
console.log("... loading");
res.on('data', function(chunk){
body += chunk;
});
res.on('end', function(){
fs.writeFileSync(filename, body);
console.log("\x1b[32m--Done--");
});
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment