Skip to content

Instantly share code, notes, and snippets.

@vpadhariya

vpadhariya/clone site using wget.txt

Created January 23, 2018 07:57
Show Gist options
  • Save vpadhariya/90ab71c2a6a1203f5f9aa75ad5c5f32a to your computer and use it in GitHub Desktop.
Save vpadhariya/90ab71c2a6a1203f5f9aa75ad5c5f32a to your computer and use it in GitHub Desktop.
Download ZIP
Clone site and remove query string values from the files in linux.
Raw
clone site using wget.txt
# Clone entire site.
wget --content-disposition --execute robots=off --recursive --no-parent --continue --no-clobber http://example.com
# Remove query string from a static resource.
for i in `find $1 -type f -name "*\?*"`; do mv $i `echo $i | cut -d? -f1`; done
@vpadhariya
Copy link
Author

OK here is the new version for 2024

Download Site Command

wget \
     --recursive \
     --level 5 \
     --no-clobber \
     --page-requisites \
     --adjust-extension \
     --span-hosts \
     --convert-links \
     --restrict-file-names=windows \
     --domains yourdomain.com \
     --no-parent \
        https://yourdomain.com

Rename files (works fine for macos)

find . -name '*\?*' | while read f; do echo mv "$f" "${f//\?*/}"; done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment