Skip to content

Instantly share code, notes, and snippets.

@konsumer
Last active November 9, 2019 11:02
Show Gist options
  • Select an option

  • Save konsumer/272e8bfe2b9f25b5ff7431068259444e to your computer and use it in GitHub Desktop.

Select an option

Save konsumer/272e8bfe2b9f25b5ff7431068259444e to your computer and use it in GitHub Desktop.
A couple common usecases for getting jquery-like for scraping, in node, from GET request
const cheerio = require('cheerio')
const fetch = require('node-fetch')
// get Authorization header for basc auth
const basicAuth = (username, password) => 'Basic ' + Buffer.from(`${encodeURIComponent(username)}:${encodeURIComponent(password)}`).toString('base64')
// get jquery-like object
const f$ = (url, options = {}) => fetch(url, options)
.then(r => r.text())
.then(r => cheerio.load(r))
const example = async () => {
const $ = await f$('https://blah')
// do jquery-type stuff here
$('a')
// basic auth
const $$ = await f$('https://blah', { headers: { authorization: basicAuth(username, password) }, credentials: 'include' })
// do jquery-type stuff here
$$('a')
}
example()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment