$ wget -e robots=off -r -np 'http://example.com/folder/'
- -e robots=off causes it to ignore robots.txt for that domain
- -r makes it recursive
- -np = no parents, so it doesn't follow links up to the parent folder
-
-
Save extratone/2d7d8509d5d9abbc30c0d694c9dedc4c to your computer and use it in GitHub Desktop.
Recursive wget ignoring robots
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment