Skip to content

Instantly share code, notes, and snippets.

@Summertime

Summertime/harvester.sh

Created October 23, 2018 12:01
Show Gist options
  • Save Summertime/7954f0031432b6b32459a7f08ee57a7b to your computer and use it in GitHub Desktop.
Save Summertime/7954f0031432b6b32459a7f08ee57a7b to your computer and use it in GitHub Desktop.
Download ZIP
Raw
harvester.sh
#!/bin/dash
seq 1000000 |
sed 's|^|https://example.com/|' |
xargs -P20 -n1000 \
curl -sSNL -X HEAD --retry 5 -w '%{http_code},%{url_effective}\n' |
tee urls.txt
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment