Notifications should be pushed out to clients (over websockets via socket.io) whenever an activity occurs that might interest them. The activities need to be routed from the activity servers (as they generate them) to the app servers who will then take care of delivering it too the clients.
- 1 Nginx Load balancer (round-robin load balancing)
- 4 App servers running socket.io
- 3 activity servers
- 2 rabbitmq servers in active/active mode
Whenever an activity is generated it gets published to a direct exchange called oae-notifications.
Messages get published with a routing key called: oae.notifications.<user id> where user id is a
string uniquely identifying the user. Messages get published without being persisted as they are transient
and have no relevance when a client is not online.
Each app server has its own queue. Whenever a client connects, a binding is made for oae.notifications.<user id>
so the messages get delivered to the correct queue. The app server can then pick off the messages and push them
out via web sockets. Whenever a client disconnects, the app server will have to remove the binding for that client
as the client could end up on another app server.
Queues don't need to be persisted either and can be removed if the connection drops (when the app server goes down)
- The amount of routes could get potentially high. (Maybe not so much of a problem with 2.4.0 trie's structure?)
- App servers are constantly adding and removing routes to the exchange.
I think this will work well. Here's a bit of feedback / random thoughts:
We might want to clarify what "oae-notifications" is, ensuring it doesn't confuse with our notifications feature. Maybe oae-push-notifications so others don't think it's an exchange dedicated for our notifications feature or something.
We should define a bit of "protocol" in the connection so it can be overloaded with different types of messages.
For example, I'm not sure strictly having
oae.notifications.<userId>binding for a client connection is enough. What if they're viewing a group's activity page? We would want to client connection to communicate that it wants to accept both notifications and that group's activities. If the user is on their own activity page, we'll want to communicate that they want to receive their own activities and notifications. What if the user is on a discussion page? We might some day want to subscribe to push messages for new messages in the discussion, etc... it would probably be better than creating 2 or more web connections for these types of things.That said, many user push connection could have multiple bindings associated to it:
Though we don't need to replicate the queues across mq nodes, our rabbitmq is currently clustered in an active-active manner, which means any app node can connect to any rabbitmq node. This means that the queues do need to be replicated to ensure a connection does miss something that happened on another node. Additionally, rabbitmq cluster is limited to using "disc" nodes only, so RAM nodes aren't an option unless we can configure a dedicated mq node for socket.io [1]. I think even disc nodes can take us a long way, though, and we can even consider having a dedicated rabbitmq ram node eventually if we start pushing its limits.
For your section on potential problems, it would be good to investigate, though bindings seem to be the optimal thing to scale out into high numbers in rabbitmq [2] if you absolutely need to. I guess in our case we would want to account for about 2 bindings per user connection (1 for notifications, one for user or group activities, or in the future messages/comments etc...).
[1] http://www.rabbitmq.com/clustering.html
[2] http://blog.springsource.org/2011/04/01/routing-topologies-for-performance-and-scalability-with-rabbitmq/