Skip to content

Instantly share code, notes, and snippets.

@sepans
Last active October 23, 2015 19:56
Show Gist options
  • Save sepans/b493a0b9d3027310e9f2 to your computer and use it in GitHub Desktop.
Save sepans/b493a0b9d3027310e9f2 to your computer and use it in GitHub Desktop.
curl ans scap an array of urls with bash script (for uk metoffice weather data)
urls=( http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/aberporthdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/armaghdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/ballypatrickdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/bradforddata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/braemardata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/cambornedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/cambridgedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/cardiffdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/chivenordata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/cwmystwythdata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/dunstaffnagedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/durhamdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/eastbournedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/eskdalemuirdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/heathrowdata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/hurndata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/lerwickdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/leucharsdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/lowestoftdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/manstondata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/nairndata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/newtonriggdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/oxforddata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/paisleydata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/ringwaydata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/rossonwyedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/shawburydata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/sheffielddata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/southamptondata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/stornowaydata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/suttonboningtondata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/tireedata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/valleydata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/waddingtondata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/whitbydata.txt
http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/wickairportdata.txt http://www.metoffice.gov.uk//pub/data/weather/uk/climate/stationdata/yeoviltondata.txt )
for (( i = 0; i < ${#urls[@]}; ++i )); do
# create filename based on url. ignore everything up to '/[a-z].txt' then remove '/'
filename=`echo ${urls[i]} | sed -e 's/.*\(\/[a-z]*.txt\)/\1/g' | sed 's/\///g'`
echo "downloading ${urls[i]} to $filename"
curl -s -o "$filename" "${urls[i]}"
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment