Hypertext Transport Protocol (HTTP) is the de facto protocol for sending data over the Internet. Today in addition to transporting hypertext documents, it is used to transport images, stylesheets, JavaScript, fonts - nearly everything in a web page. Rarely do developers pause to question why this is. Lately, I've been thinking a lot about HTTP and its relation to APIs and web apps. This document contains my thoughts on the usefulness of HTTP requests as envelopes for sending data.
HTTP requests consist of at least 7 pieces: the path, the query string, the method, the status, the headers, the body, and cookies. (Technically cookies are a subcategory of headers but they have additional behaviors that make them worth examining separately.) This article examines each of them from the perspective of their utility for carrying arbitrary data. Note: This article only looks at HTTP/1.
REST APIs often make use of HTTP methods and response status codes to convey information. I would classify these as ways of encoding enumerated types. Often this is supplemented by data in the HTTP response body, which is because these mechanisms are insufficient to communicate anything substantial.
| Availability | Pros | Cons | |
|---|---|---|---|
| HTTP method | client | Shown prominently in DevTools. | Need special tools like curl or Postman to send data. |
| HTTP status codes | server | Well organized (200, 404, 500, etc). Shown prominently in DevTools. | No way to include extra data including error details. Unidirectional (set by server). |
HTTP provides a number of ways to encode key / value pairs in requests and responses. Key/value stores (Redis, IndexDB, LevelDB, etc) have had a resurgence of popularity of late as a NoSQL alternative to relational databases. These encodings might be an elegant match for some use cases.
| Availability | Pros | Cons | |
|---|---|---|---|
| HTTP X-Custom-Headers | server & client | None? | Need special tools to render and parse. No theoretical size limit, but a practical limit of ~10,000 characters. |
| Cookies | server & client | Flexible. Can be read/writeable in JS, or hidden from JS, and can force an encrypted HTTPS-only transport. | Poor separation of concerns. Variously acts as an arbitrary client-local storage, as a client identification system (and hence security mechanism), in addition to its ability to encode data. |
| URL query parameters | client | Highly accessible. Can be typed directly into the address bar. | More complex data (whitespace) requires awkward variant of percent-encoding. No theoretical size limit, but a practical limit in most software of ~1,000 characters. |
Lastly, HTTP provides a few ways that work well for encoding arbitrary data. Technically, any of the above methods can encode arbitrary data. (For example, you could return any number in the HTTP status code, and thus treat the status code as a bit-string.) However, doing so eliminates the advantage of using HTTP in the first place - the availability of parsing, rendering, and debugging tooling already present in the software ecosystem (or even just in the browser DevTools).
| Availability | Pros | Cons | |
|---|---|---|---|
| URL path | client | Highly accessible. Can be typed directly into the address bar. Works well for some hierarchical data. | Poor separation of concerns. The domain is always a fixed address specifier due to DNS. The fragment identifier can represent client state such as scrollbar position. Only the path part of the URL can encode arbitrary data, and even then it must be percent-encoded. |
| HTTP body | server & client | Extremely flexible. Practically no size limit. Allows true arbitrary data. (Encoding is specified in HTTP header.) | Only available with PUT and POST. No well-established semantic convention for structuring responses. The variety of encodings (application/x-www-form-urlencoded, multipart/form-data, application/json) can be overwhelming. |
There are at least 7 distinct ways to encode information:
| Name | Examples | |
|---|---|---|
| 1. | URL path | (/users, /user/1, /user/1/msgs, /user/1/msg/1, ...) |
| 2. | URL query | (?id=1, ?q=search&order=desc, ?page=2, ?array=foo&array=bar, ...) |
| 3. | HTTP method | (GET, PUT, POST, DELETE, ...) |
| 4. | HTTP status | (200 OK, 404 Not Found, 201 Created, ...) |
| 5. | HTTP headers | (Authorization: Bearer $TOKEN, X-RateLimit-Remaining: 56, ...) |
| 6. | HTTP body | (multipart/form-data, XML, JSON, ...) |
| 7. | HTTP cookies | (connect.sid, PHPSESSID, ...) |
Most REST APIs actually utilize most of these features, perhaps skipping over query parameters and cookies. This wouldn't be a big deal if using each of the ways was easy and consistent. However in practice, each way has its own API and set of quirks that have to be remembered. Consider jQuery.ajax:
| Name | How to use |
|---|---|
| URL path | specified as 1st argument |
| URL query | use data setting with GET requests |
| HTTP method | use method setting OR more frequently use a helper function ( |
| HTTP status | use separate callbacks for "errors" and "successes" |
| HTTP headers | use headers setting |
| HTTP body | use data setting with appropriate dataType setting |
| HTTP cookies | use xhrFields.withCredentials: true setting to include existing cookies in requests.* |
* However, there is no way to edit the cookie (or read it) using just jQuery.
I put forward the hypothesis that the HTTP request protocol is probably more complicated than it needs to be. This added complexity means there are more things that can go wrong, which means there are more things developers need to know about to debug their code. This, in turn, steepens the learning curve and slows down development.