Let me explain what we're trying to do, which may be crazy. But if it's not crazy, perhaps someone can tell us how to use AMQP to do it.
We have a Rails app that talks to a growing number of backend Clojure services.
It could talk to them via HTTP, but many of the messages don't need to be synchronous. They're "fire and forget". Hence AMQP.
For the synchronous messages, we observed two things:
-
Serializing and deserializing HTTP is surprisingly annoying, especially given Rails conventions.
-
Most of the time when we send a message that updates a resource, we don't need a synchronous acknowledgement right away. We only need to know that the update has happened at some point before we next read the resource.
We already use Redis for caching, and Redis has these nice blocking queues/lists. So what if…
-
Instead of sending a synchronous HTTP POST (for example), we send an AMQP fire-and-forget message. (Note: this seems better than Redis publish/subscribe because AMQP queues make it easy to distribute the load among N workers.)
-
The message includes the name of a blocking Redis queue. The resource-owner should push a value on that queue when it's finished the update.
-
Any time the Rails client needs to access resource information (by fetching it from Redis), it first waits on the blocking queue (which will almost always already be unblocked).
-
Profit! Efficiency!
Early indications are that this works well when the change to the resource is in Request N and the next access is in Request N+m. But it happens that I upgraded a controller that both changes a resource in Request N and later blocks on the same Resource. Because the AMQP message doesn't go out until the end of the Request/Response cycle, the controller blocked for the entire timeout period waiting to be told that its resource was ready.
@marick, a couple of questions: