$ wget -e robots=off -r -np 'http://example.com/folder/'
- -e robots=off causes it to ignore robots.txt for that domain
- -r makes it recursive
- -np = no parents, so it doesn't follow links up to the parent folder
-
-
Save shaneholloman/0eab95e09d67c97d98c37ee5644a83bb to your computer and use it in GitHub Desktop.
Recursive wget ignoring robots
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment