DanNi0130

DanNi0130 / gist:0f959de6f41839a1b45328a33daabb8e

Last active February 18, 2019 06:14

npm install --save request request-promise cheerio puppeteer

DanNi0130 / install libraries

Last active February 18, 2019 06:15

npm install --save request request-promise cheerio puppeteer

DanNi0130 / potusScraper.js v1

Last active February 18, 2019 06:15

	const rp = require('request-promise');
	const url = 'https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States';

	rp(url)
	.then(function(html){
	//success!
	console.log(html);
	})
	.catch(function(err){
	//handle error

DanNi0130 / potusScraper.js v1 output

Created July 19, 2018 01:21

	<!DOCTYPE html>
	<html class="client-nojs" lang="en" dir="ltr">
	<head>
	<meta charset="UTF-8"/>
	<title>List of Presidents of the United States - Wikipedia</title>
	...

DanNi0130 / potusScraper.js v2

Created July 19, 2018 01:24

	const rp = require('request-promise');
	const $ = require('cheerio');
	const url = 'https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States';

	rp(url)
	.then(function(html){
	//success!
	console.log($('big > a', html).length);
	console.log($('big > a', html));
	})

DanNi0130 / potusScraper.js v2 output

Created July 19, 2018 01:25

	45
	{ '0':
	{ type: 'tag',
	name: 'a',
	attribs: { href: '/wiki/George_Washington', title: 'George Washington' },
	children: [ [Object] ],
	next: null,
	prev: null,
	parent:
	{ type: 'tag',

DanNi0130 / potusScraper.js v3

Created July 19, 2018 01:26

	const rp = require('request-promise');
	const $ = require('cheerio');
	const url = 'https://en.wikipedia.org/wiki/List_of_Presidents_of_the_United_States';

	rp(url)
	.then(function(html){
	//success!
	const wikiUrls = [];
	for (let i = 0; i < 45; i++) {
	wikiUrls.push($('big > a', html)[i].attribs.href);

DanNi0130 / potusScraper.js v3 output

Created July 19, 2018 01:27

DanNi0130 / potusParse.js v1

Created July 19, 2018 01:29

	const rp = require('request-promise');
	const url = 'https://en.wikipedia.org/wiki/George_Washington';

	rp(url)
	.then(function(html) {
	console.log(html);
	})
	.catch(function(err) {
	//handle error
	});

DanNi0130 / potusParse.js v1 output

Created July 19, 2018 01:30

	<html class="client-nojs" lang="en" dir="ltr">
	<head>
	<meta charset="UTF-8"/>
	<title>George Washington - Wikipedia</title>
	...