Skip to content

Instantly share code, notes, and snippets.

@abhiomkar

abhiomkar/wget_urls.sh

Created August 29, 2011 11:43
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save abhiomkar/1178236 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save abhiomkar/1178236 to your computer and use it in GitHub Desktop.
Download ZIP
Recursively Fetch all the possible URLs
Raw
wget_urls.sh
wget -R.jpg,.jpeg,.gif,.png,.css,.js -c -r http://www.example.com/ -o urllog.txt
grep ' http' urllog.txt | awk '{print $3}'
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment