Skip to content

Instantly share code, notes, and snippets.

@dpedu

dpedu/gist:6920a29fc4da2e9372ef4c71a6739719

Created April 25, 2017 04:17
Show Gist options
  • Save dpedu/6920a29fc4da2e9372ef4c71a6739719 to your computer and use it in GitHub Desktop.
Save dpedu/6920a29fc4da2e9372ef4c71a6739719 to your computer and use it in GitHub Desktop.
Download ZIP
parallel wget open directory mirroring
Raw
gistfile1.txt
#!/bin/bash -ex
URL="http://example.com/open/directory/"
WGETCMD="wget -nv -e robots=off --recursive -nc -np --recursive --level inf --limit-rate=1m $URL"
PARALLELS=7
time ( seq $PARALLELS | parallel -j$PARALLELS -n1 bash -c "\"sleep \$(shuf -i 1-30 -n 1) ; $WGETCMD --append-output=logs/wget-{}.log\"" )
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment