Skip to content

Instantly share code, notes, and snippets.

@z010107
Forked from vpadhariya/clone site using wget.txt
Created August 30, 2018 14:10
Show Gist options
  • Save z010107/fbb4b643401e6a027162d47ea9fabc08 to your computer and use it in GitHub Desktop.
Save z010107/fbb4b643401e6a027162d47ea9fabc08 to your computer and use it in GitHub Desktop.
Clone site and remove query string values from the files in linux.
# Clone entire site.
wget --content-disposition --execute robots=off --recursive --no-parent --continue --no-clobber http://example.com
# Remove query string from a static resource.
for i in `find $1 -type f -name "*\?*"`; do mv $i `echo $i | cut -d? -f1`; done
@z010107
Copy link
Author

z010107 commented Aug 30, 2018

# /bin/bash
for i in `find $1 -type f -name "*\?*"`; 
  do mv $i `echo $i | cut -d? -f1`; 
done

@z010107
Copy link
Author

z010107 commented Sep 18, 2018

Maybe like this - wget --content-disposition -p -k https://www.oasiscatalog.com/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment