Let’s hypothesize that I have the magical ability to open a connection to any web server, and that every key press I make on my keyboard will be sent to the remote computer. I want to get the Google homepage, so I connect to one of their many web servers.
If the server spoke English, I might type:
Hello! Please give me the content located at the path /
on the domain www.google.com.
I'm using the Chrome web browser, version 26.
I would like to receive the content as HTML,
zipped to save bandwidth,
in US English,
with UTF-8 characters (like a snowman ☃).
This is my only request.
Of course, computers prefer structured data since it’s easier to parse. What HTTP does is give me a way to convey the above information (and more, if I wanted) in a manner which any HTTP-compatible server will be able to understand. In fact, the exact same request is expressed as the following in version 1.1 of the HTTP protocol:
GET / HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.65 Safari/537.31
Accept: text/html
Accept-Encoding: gzip
Accept-Language: en-US
Accept-Charset: utf-8
Connection: close
And that’s it! All I’d have to do is open up my hypothetical program which would let me type to a web server, type the lines above, and pipe the response to a file or other destination (probably a decompression utility since the response would likely be compressed). The great thing about HTTP is that it’s human-readable. It’s probably as low as one can go in the whole web stack and still have a chance of constructing a query and response from hand. Which is, of course, why I think that understanding raw HTTP is still a reasonable requirement to be expected of any programmer intent on using web services.
And that hypothetical program which allows me to type out HTTP requests? It’s called Netcat, and it’s awesome for testing HTTP servers. If you’re lucky enough to use a computer with a decent command line, you probably already have this installed, and if you use Windows, there are ports available too.
If you’re on Linux, Unix or OSX, try the following from a command prompt:
nc maps.googleapis.com 80
That opens a connection to a Google Maps API server on port 80, which is not-coincidentally, the standard port for HTTP requests. Everything you type after pressing enter is sent to the server, and everything sent back is printed to the console (there is no other output). Since the server won’t send anything until it gets a request, the ball is in your court. Copy and paste:
GET /maps/api/geocode/json?address=San+Francisco,+CA&sensor=false HTTP/1.1
Host: maps.googleapis.com
Connection: close
Then press enter twice and watch the magic happen. You are now COMMUNICATING WITH MACHINES. The response will be a JSON-encoded representation of how Google treats an address of “San Francisco, CA”.
You can pretty much play with any URL you’d access in your browser in this way. One thing to keep in mind is that if the url begins with https (note the trailing “s”) the connection needs to be encrypted, which Netcat isn’t so good at.