Skip to content

Instantly share code, notes, and snippets.

@dzogrim
Last active February 14, 2019 15:14
Show Gist options
  • Select an option

  • Save dzogrim/ebf96395b40c6ae3d441b04eb2430626 to your computer and use it in GitHub Desktop.

Select an option

Save dzogrim/ebf96395b40c6ae3d441b04eb2430626 to your computer and use it in GitHub Desktop.
Parallelize many curl requests and return OK or not
#!/bin/bash
set -x -e
expectedCode=200
log_file="/tmp/$(basename $0).log"
input_links="${HOME}/Documents/redirections.url.txt"
mycurl(){
redirection="$1"
[[ "$( curl -s --location -o /dev/null -w "%{http_code}" "${redirection}" )" -ne "${expectedCode}" ]]
}
export -f mycurl
[[ ! $( which parallel ) ]] && exit 1
[[ ! -f ${input_links} ]] && exit 1
parallel --jobs 16 --halt soon,success=90% mycurl < ${input_links} > ${log_file} 2>&1
if [ $? != 0 ]
then #Exiting with error
echo 1
else #Exiting normally
echo 0
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment