simonw/wget.md

Created December 9, 2016 06:38

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/simonw/27e810771137408fd7834ad153750c41.js"></script>
Save simonw/27e810771137408fd7834ad153750c41 to your computer and use it in GitHub Desktop.

Recursive wget ignoring robots

Raw

    $ wget -e robots=off -r -np 'http://example.com/folder/'

-e robots=off causes it to ignore robots.txt for that domain
-r makes it recursive
-np = no parents, so it doesn't follow links up to the parent folder

taoyichen commented Jan 14, 2020

wget -e robots=off -r -np --page-requisites --convert-links
For websites

tumelo-mapheto commented Jun 25, 2020

Thanks for sharing.

fsiler commented Feb 28, 2021

I'm still getting no-follow attribute found in $URL. Will not follow any links on this page after using wget -e robots=off -r -np --page-requisites --convert-links $SITE. Is this a bug?

NilsIrl commented Apr 16, 2021 •

edited

Loading

I'm still getting no-follow attribute found in $URL. Will not follow any links on this page after using wget -e robots=off -r -np --page-requisites --convert-links $SITE. Is this a bug?

Yes, this is a bug, it should be fixed in the next version of wget: https://git.savannah.gnu.org/cgit/wget.git/commit/?id=f1cccd2c454fb416e75a22b358b0a11266642007

See https://www.reddit.com/r/DataHoarder/comments/mprq89/wget_respects_nofollow_attribute_despite_e/guct2s5/ for more details