THEORY: Little's Law and Applying Back Pressure When Overloaded

littles-law-and-back-pressure.md

Applying Back Pressure When Overloaded

[...]

Let’s assume we have asynchronous transaction services fronted by an input and output queues, or similar FIFO structures. If we want the system to meet a response time quality-of-service (QOS) guarantee, then we need to consider the three following variables:

1. The time taken for individual transactions on a thread
2. The number of threads in a pool that can execute transactions in parallel
3. The length of the input queue to set the maximum acceptable latency

max latency  = (transaction time / number of threads) * queue length
queue length = max latency / (transaction time / number of threads)

By allowing the queue to be unbounded the latency will continue to increase. So if we want to set a maximum response time then we need to limit the queue length.

By bounding the input queue we block the thread receiving network packets which will apply back pressure up stream. If the network protocol is TCP, similar back pressure is applied via the filling of network buffers, on the sender. This process can repeat all the way back via the gateway to the customer. For each service we need to configure the queues so that they do their part in achieving the required quality-of-service for the end-to-end customer experience.

One of the biggest wins I often find is to improve the time taken to process individual transaction latency. This helps in the best and worst case scenarios.

[...]

RabbitMQ: Stack Overflow Behavior

RabbitMQ's "stack overflow behavior" refers to how the system handles situations where a queue's capacity is exceeded, leading to an "overflow" of messages. This is particularly relevant when the rate of messages being published to a queue is significantly higher than the rate at which consumers can process them.

---

Key Aspects of RabbitMQ Queue Overflow Behavior

📏 Maximum Queue Length

RabbitMQ allows configuring a max-length for queues. This limit dictates the maximum number of messages a queue can hold. Setting this limit is crucial for preventing queues from consuming excessive memory and resources.

---

⚙️ Overflow Strategies

When a queue reaches its max-length, RabbitMQ employs a defined overflow strategy to determine how new messages are handled:

• drop-head (Default)
  This strategy, the default for classic queues, discards the oldest message in the queue when a new message arrives and the queue is full. This ensures new messages are accepted while maintaining the queue's length limit.

• reject-publish
  With this strategy, newly published messages are rejected (nacked) by the broker when the queue is full. This signals to the publisher that the message could not be enqueued.

• reject-publish-dlx
  Similar to reject-publish, but rejected messages are routed to a Dead Letter Exchange (DLX) if configured. This allows for specific handling of rejected messages, such as logging or retrying.

---

🚦 Flow Control

Beyond explicit overflow strategies, RabbitMQ implements a credit flow control mechanism. This system monitors the rate of message consumption and can temporarily block publishers if consumers are falling significantly behind, preventing the broker from becoming overwhelmed and potentially crashing due to memory exhaustion.

Note: This is distinct from queue overflow but contributes to overall system stability.

---

⏱️ Message TTL (Time-To-Live)

While not an overflow strategy in itself, setting a message-ttl can help manage queue size by automatically expiring messages after a specified time, regardless of whether they have been consumed. This can prevent queues from growing indefinitely due to slow consumers or unconsumed messages.

---

⚠️ Consequences of Overflow

• Message Loss: If drop-head is used, older messages are discarded.
• Publisher Blocking/Errors: If reject-publish or flow control is active, publishers may experience delays or receive errors when attempting to send messages.
• Resource Consumption: Without proper limits and strategies, an overflowing queue can consume excessive memory on the RabbitMQ server, potentially impacting performance or leading to crashes.

---

✅ Addressing Queue Overflow

• Increase Consumer Capacity
  Scale up the number of consumers or optimize consumer processing logic to handle messages more efficiently.

• Optimize Queue Length
  Adjust the max-length based on system requirements and expected message volume.

• Implement Overflow Strategies
  Choose the appropriate x-overflow strategy (for classic queues) based on desired message handling behavior.

• Utilize Message TTL
  Set TTLs for messages or queues to prevent indefinite message accumulation.

• Monitor and Alert
  Implement monitoring to detect queue overflow conditions and trigger alerts for proactive intervention.