Puppeteer Resources

puppeteer-boilerplate.js

const puppeteer = require("puppeteer");

let browser;
(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  const ua =
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36";
  await page.setUserAgent(ua);
  const url = "https://www.example.com";

  // optional optimization stubs:
  //await page.setJavaScriptEnabled(false);
  //await page.setRequestInterception(true);
  //const allowed = [
  //  "https://www.example.com",
  //];
  //page.on("request", request => {
  //  if (allowed.some(e => request.url().startsWith(e))) {
  //    request.continue();
  //  }
  //  else {
  //    request.abort();
  //  }
  //});

  //page.on("request", req => {
  //  if (req.resourceType() === "image") {
  //    req.abort();
  //  }
  //  else {
  //    req.continue();
  //  }
  //});

  await page.goto(url, {waitUntil: "domcontentloaded"});
  const $ = (...args) => page.waitForSelector(...args);
  console.log(await page.title());
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

puppeteer-resources.md

Puppeteer Resources

• Logging JSON to the console

  page.on("console", msg =>
    Promise.all(msg.args().map(e => e.jsonValue()))
      .then(args => console.log(...args))
  );

• Registering .on handlers on a page
• Waiting for responses with page.waitForResponse
• Mocking repsonses and mocking fetch in browser context
• Navigating a SPA
• Promisifying page.on("request", ...)
• Opening a bunch of links -- can use browser.newPage() to create new tabs
• Persisting browser data:
  • Storing previous sessions/browser user data
  • How to manage log in session through headless chrome?
• Filling out forms inside iframes
  • Clicking on an element in an iframe fails
  • Clicking element inside iFrame does not give any response
• Injecting functions into the browser:
  • How to pass a function in Puppeteers .evaluate() method?
  • How can I dynamically inject functions to evaluate using Puppeteer?
• Disable JS: await page.setJavaScriptEnabled(false);
• Disable images (can also block other resource types such as "stylesheet"):

  await page.setRequestInterception(true);
  page.on("request", req => {
    if (req.resourceType() === "image") {
      req.abort();
    }
    else {
      req.continue();
    }
  });

• Puppeteer can't find elements when Headless TRUE

Gotchas

• Use the Promise.all pattern when triggering navigation.
• Make sure to use page.waitForSelector("...", {visible: true}) before clicking an element when getting false negatives.
• If page.waitForSelector isn't firing or page.click has no effect, it could be that there is a hidden element with the same selector earlier in the page than the targeted one. Puppeteer is waiting for the wrong element to appear, basically. Another possible scenario is the element has no text or is zero width. See Puppeteer in NodeJS reports 'Error: Node is either not visible or not an HTMLElement'. Similarly, using eval and a native visibility check can help bypass {visible: true} and page.$() false negatives.
• page.exposeFunction runs the function in Node context, not browser context. See: Why can't I access 'window' in an exposeFunction() function with Puppeteer?
• Detecting navigation end can be tricky and inconsistent. See Puppeteer wait until page is completely loaded.
• Using page.click can change the history push behavior relative to triggering a navigation with page.evaluate(). See this.
• For some sites, headless: false works but headless: true fails. See this thread and this SO answer.

"load" can time out

On a site that requests a resource that has a long 504 gateway timeout, "load" (the default nav predicate) can time out but "domcontentloaded" won't:

const puppeteer = require("puppeteer"); // ^22.7.1

// request some resource with a 504 gateway timeout
const html = `<img src="https://via.placeholder.com/150/FF0000/FFFFFF?text=CD">`
let browser;

(async () => {
  browser = await puppeteer.launch();
  const [page] = await browser.pages();
  await page.setContent(html, {
    waitUntil: "load" // fix with: "domcontentloaded"
  }); // times out on "load"
})()
  .catch(err => console.error(err))
  .finally(() => browser?.close());

Unresolved

• Puppeteer flow logic, check if navigation has occurred (vs wait for)

Haven't used yet but interesting

• Preventing Puppeteer Detection
• Headless file downloads
• Another file download resource
• How to do a POST request in Puppeteer