Fully working crawler using wget

crawl.sh

wget -nv -t3 -c -nH -r -l0 -k -p -np -e robots=off --reject-regex "\/\?(.*)" [url]
 

crawl_detailed.sh

wget \
  --no-verbose \
  --tries=3 \
  --continue \
  --retry-connrefused \
  --no-host-directories \
  --recursive \
  --level 0 \
  --convert-links \
  --page-requisites \
  --no-parent \
  -e robots=off \
  --reject-regex "\/\?(.*)" \
  [url]
   

fix_names.sh

for f in `find . -regex ".*?.*"`; do mv -v $f $(echo $f | sed "s/\?.*//") ; done
 

fix_names_detailed.sh

for f in `find . -regex ".*?.*"`;
  do mv -v $f $(echo $f | sed "s/\?.*//")
done
 

Some notes:

• For an IE fix, stylesheets usually contain fonts with a fake query param like fontawesome-webfont.eot?v=4.2.0 We need to allow wget crawling such URLs too but prevent directory listing sort params. So reject-regex is more open. If you want to strictly prevent crawling any url with params use "(.*)\?(.*)". fix_names.sh helps removing query from file names.

• --adjust-extension or -E flag is optional. Can be useful for sites that use friendly URLs without .html extension. But use with caution as it may also have wrong behaviour like adding .html to css files!