Downloading Dynamic Webpages With Chrome and Monolith

dynamicpagedownload.md

The following is a set of instructions for downloading a static or dynamic webpage into a single .html file.

Tools

1. Download and install Monolith: https://github.com/Y2Z/monolith
   • If on Windows, I recommend using winget: winget install --id=Y2Z.Monolith -e
2. Download and install Chrome if you don't currently use it.
   • Find the path to Chrome.exe on your machine, it should look something like: C:\Users\[user]\AppData\Local\Google\Chrome\Application
   • Add this to the Path variable in Windows, guide: https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/

With Monolith and Chrome installed, you should have access to their executables via command line (PowerShell recommended for Windows).

You can test both by trying them out in PowerShell:

• monolith --version -> This will print the version of the Monolith CLI.
• chrome -> This will open a chrome window if it's working.

Process

If you're downloading a static page that doesn't have any dynamically loading content (content that loads after the initial webpage loads), you can use Monolith by itself:

monolith http://eternal-city.wikidot.com/missile-weapons-bows -o missile-weapons-bows.html

If you're after a page that has dynamic content, you'll have to first open a headless Chrome window and pipe that content into Monolith:

chrome --headless --window-size=1920,1080 --run-all-compositor-stages-before-draw --virtual-time-budget=9000 --incognito --dump-dom http://eternal-city.wikidot.com/rank-bonus-calculator | monolith - -I -b http://eternal-city.wikidot.com/rank-bonus-calculator -o rank-bonus-calculator.html

PowerShell Script

Here is a set of PowerShell scripts to help automate this: https://github.com/jdalley/page-downloader