Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Select an option

  • Save innermond/ab67782ad0e09d8dc52f to your computer and use it in GitHub Desktop.

Select an option

Save innermond/ab67782ad0e09d8dc52f to your computer and use it in GitHub Desktop.
Download with wget using a .list of urls. Every download happens in parallel and every .list has it's own process
#!/bin/bash
path=$(realpath ${1:-.})
parallel=${2:-8}
directory=
for list in $(find "$path" -wholename "*/.list")
do
# .ignore file exists?
directory="${list%/*}"
if [ -f "$directory"/.ignore ]; then
continue
fi
cd "$directory"
( cat $list | xargs -I % -n 1 -P $parallel wget -t 1 --header="Accept: text/html" --user-agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0" ‐‐referer="https://www.google.com" % ) &
cd "$path"
done
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment