Skip to content

Instantly share code, notes, and snippets.

@pixelbrackets

pixelbrackets/bash

Last active November 10, 2023 09:05
Show Gist options
  • Save pixelbrackets/6c18175b7b3528d4702dd71e325390d0 to your computer and use it in GitHub Desktop.
Save pixelbrackets/6c18175b7b3528d4702dd71e325390d0 to your computer and use it in GitHub Desktop.
Download ZIP
wget archive
Raw
bash
### Mirror page (infinite recursion), download assets as well, add html extension to pagelinks, make links relative, wait one second after each request to avoid hitting rate limits, identify as archive crawler
wget --mirror --page-requisites --adjust-extension --convert-links --wait=1 --user-agent="Archive Crawler" https://www.example.com
### maybe you want/need to turn off checks for robots.txt as well: `--execute robots=off`
### Watch out for relative/absolute links, with or without www. Crawling »www.example.com«, but all links point to »example.com« will cause wget to mirror the first page URL ony, and not follow any more links.
### To allow multipe domains use `--span-hosts --domains=example.com,www.example.com,cdn.example.com`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment