techie2000/wget.sh

Last active January 24, 2021 16:32

Star () You must be signed in to star a gist
Fork () You must be signed in to fork a gist

Learn more about clone URLs
Clone this repository at <script src="https://gist.github.com/techie2000/a6568ab68be28aad77f578d7fe548f56.js"></script>
Save techie2000/a6568ab68be28aad77f578d7fe548f56 to your computer and use it in GitHub Desktop.

Download ZIP

Download a site

Raw

wget.sh

wget --mirror --no-parent --continue --limit-rate=512k --wait=0 --reject "index.html*, foo.txt, bar*.*" "https://the-eye.eu/public/AppleArchive/"

Author

techie2000 commented Jan 24, 2021

Consider adding --random-wait to try and prevent blocking by sits that analyse logs and try to spot/prevent wget type ops

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment