Skip to content

Instantly share code, notes, and snippets.

@brovish
Forked from crittermike/wget.sh
Last active June 16, 2021 17:43
Show Gist options
  • Select an option

  • Save brovish/d2595264d7bc2b0a384deee5e70c62dd to your computer and use it in GitHub Desktop.

Select an option

Save brovish/d2595264d7bc2b0a384deee5e70c62dd to your computer and use it in GitHub Desktop.
Download an entire website with wget, along with assets.
# One liner
wget --recursive --page-requisites --adjust-extension --span-hosts --convert-links --restrict-file-names=windows --domains yoursite.com --no-parent yoursite.com
# Explained
wget \
--recursive \ # Download the whole site.
--page-requisites \ # Get all assets/elements (CSS/JS/images).
--adjust-extension \ # Save files with .html on the end.
--span-hosts \ # Include necessary assets from offsite as well.
--convert-links \ # Update links to still work in the static version.
--restrict-file-names=windows \ # Modify filenames to work in Windows as well.
--domains yoursite.com \ # Do not follow links outside this domain.
--no-parent \ # Don't follow links outside the directory you pass in.
yoursite.com/whatever/path # The URL to download
@echo off
start "" /d F:\onedrive\books_sql wget --mirror -p --html-extension --convert-links -e robots=off -P . https://sqlserverperformace.blogspot.com/
@echo off
start "" /d F:\onedrive\books_sql\ wget --wait=2 --level=inf --recursive --page-requisites --user-agent=Mozilla --no-parent --convert-links --adjust-extension --no-clobber -e robots=off http://www.example.com/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment