Skip to content

Instantly share code, notes, and snippets.

@aaronlidman

aaronlidman/gist:3717548

Created September 13, 2012 20:49
Show Gist options
  • Save aaronlidman/3717548 to your computer and use it in GitHub Desktop.
Save aaronlidman/3717548 to your computer and use it in GitHub Desktop.
Download ZIP
get a list of links
Raw
gistfile1.txt
# remove the -v on the last grep to change from blacklist to whitelist
wget --spider --force-html -r -l2 http://example.com 2>&1 | grep '^--' | awk '{ print $3 }' | grep -v '\.\(css\|js\|png\|gif\|jpg\)$' > wget.out
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment