Skip to content

Instantly share code, notes, and snippets.

@jlongman
Last active August 29, 2015 14:20
Show Gist options
  • Save jlongman/048fd1a6d3b5a61067f0 to your computer and use it in GitHub Desktop.
Save jlongman/048fd1a6d3b5a61067f0 to your computer and use it in GitHub Desktop.
http/1.1 client one liner
export USER_AGENT="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:27.0) Gecko/20100101 Firefox/27.0"
http10get() {
python -c "import urllib; urllib.urlretrieve(\"$1\")"
}
http11get () { python2.7 -c "from urlparse import urlparse
import os
import urllib2
path = \"$XIPIX_DIR\" + urlparse(\"$1\").path;
try:
os.makedirs(os.path.dirname(path));
except:
pass
try:
request = urllib2.Request(\"$1\", headers={ \"User-Agent\": \"${USER_AGENT}\"})
data = urllib2.urlopen(request).read()
f = open(path, \"w\");
f.write(data)
f.close()
except Exception as e:
print \"Error with $1\"
raise e
"; }
# usage:
http11get http://www.foo.com
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment