THEORY: Little's Law and Applying Back Pressure When Overloaded

littles-law-and-back-pressure.md

Applying Back Pressure When Overloaded

[...]

Let’s assume we have asynchronous transaction services fronted by an input and output queues, or similar FIFO structures. If we want the system to meet a response time quality-of-service (QOS) guarantee, then we need to consider the three following variables:

1. The time taken for individual transactions on a thread
2. The number of threads in a pool that can execute transactions in parallel
3. The length of the input queue to set the maximum acceptable latency

max latency  = (transaction time / number of threads) * queue length
queue length = max latency / (transaction time / number of threads)

By allowing the queue to be unbounded the latency will continue to increase. So if we want to set a maximum response time then we need to limit the queue length.

By bounding the input queue we block the thread receiving network packets which will apply back pressure up stream. If the network protocol is TCP, similar back pressure is applied via the filling of network buffers, on the sender. This process can repeat all the way back via the gateway to the customer. For each service we need to configure the queues so that they do their part in achieving the required quality-of-service for the end-to-end customer experience.

One of the biggest wins I often find is to improve the time taken to process individual transaction latency. This helps in the best and worst case scenarios.

[...]

Amazon Builders' Library

Some good articles from Amazon Builders' Library about handling overloaded systems and back-pressure mechanisms.

• Using load shedding to avoid overload
• Avoiding overload in distributed systems by putting the smaller service in control
• Avoiding insurmountable queue backlogs

⭐️ Retry strategies and their impact on overloaded systems

⭐️⭐️ Good Retry, Bad Retry: An Incident Story

This article is gold! It shows how some retry techniques might overload a system through a DIDACTIC and well-written story. It covers techniques such as:

1. Simple retry;
2. Retry with backoff;
3. Retry with backoff and jitter;
4. Retry circuit breaker: The service client completely disables retries if the percentage of service errors exceeds a certain threshold (for example, 10%). As soon as the percentage of errors within an arbitrary minute drops below the threshold, retries are resumed. If the service experiences problems, it won’t receive any additional load from retries;
5. Retry budget (or adaptive retry): Retries are always allowed, but within a budget, for example, no more than 10% of the number of successful requests. In case of service problems, it can receive no more than 10% of additional traffic;
6. Retry + Circuit breaker(threshold=10%);
7. Retry + Circuit breaker(threshold=50%);
8. Retry + Deadline propagation;

Both (Retry circuit breaker and Retry budget) options guarantee that in case of service problems, clients will add no more than n% of additional load to it

[...] it’s necessary to differentiate between scenarios when the service is healthy and when it’s experiencing problems. If the service is healthy, it can be retried because errors might be transient. If the service is having issues, retries should be stopped or minimized.

The percentage of retries can be calculated locally without complicating the system with global statistics synchronization.

Ben conducted a simulation: for long-lived clients, local statistics behave identically to global ones, and exponential backoff doesn’t significantly impact amplification.

Based on these findings, Ben decided to propose a new postmortem action item: implementing a retry budget with a 10% limit, in addition to the existing exponential backoff. There’s no need for global statistics synchronization — a local token bucket should be enough.

References that are worth reading

• ⭐️ Google SRE Book: Handling Overload
• ⭐️ Marc's blog: Open and Closed, Omission and Collapse
  • Wiki - Control loop: Open-loop and closed-loop
  • Wiki - Negative feedback
  • Wiki - Control engineering
• ⭐️ Marc's blog: Fixing retries with token buckets and circuit breakers
• ⭐️ Marc's blog: What is Backoff For?

  • The only way to deal with long-term overload is to reduce load, deferring load does not work.

  • A good approach to retries combines backoff, jitter, and a good retry policy. These are complimentary mechanisms, and neither solves the whole problem.

• ⭐️ Marc's blog: Will circuit breakers solve my problems?

  Modern distributed systems are designed to partially fail, continuing to provide service to some clients even if they can’t please everybody. Circuit breakers are designed to turn partial failures into complete failures. One mechanism will likely defeat the other.

• ⭐️ Marc's blog: Metastability and Distributed Systems
  • Metastable Failures in Distributed Systems
• Marc's blog: Simple Simulations for System Builders
  • Marc's blog: Serial, Parallel, and Quorum Latencies

Uma forma simples de lembrar:

• Open/Closed responde: "Existe feedback?"
• Positive/Negative responde: "O feedback corrige ou amplifica?"

Open-loop  -> sem feedback
Closed-loop -> com feedback

Closed-loop
    ├─ Negative feedback (estabiliza)
    └─ Positive feedback (amplifica)

É por isso que, em sistemas distribuídos, quase todos os mecanismos de resiliência (autoscaling, congestion control, adaptive throttling, backpressure) são essencialmente closed-loop negative feedback controllers.

Annotations (pt_BR)

Esse artigo eh PERFEITO, gesuis! 🤩🤩🤩
https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6

O artigo eh sobre como retries podem sobrecarregar seu sistema e como lidar com isso.

Resumo do resumo:

Retries são perigosos, isso já sabemos. Mas como estratégias de retry impactam negativamente na sobrecarga do sistema eh onde fica interessante.

O artigo testa algumas estrategias de retry em alguns cenarios através de simulações. Mas como eh pra resumir o artigo que eh longo, vamos lá...

A estratégia de Retry+backoff+jitter funciona muito bem para sistemas que sejam considerados saudáveis (healthy), ou seja, que estão enfrentando uma sobrecarga temporaria, indisponibilidade parcial, mas principalmente curta, ou seja, que causa transient errors, mas ela não é de muita ajuda em sobrecargas longas (particionamento de rede, crash da aplicação ou alta taxa de erro), pois ela apenas posterga a sobrecarga da aplicação, aumentando o tempo de recovery da aplicação. De forma direta, podemos inferir que, se o tempo de sobrecarga for superior ao tempo que os clients (que fazem retry) estão dispostos a esperar, então os retries estão apenas piorando a situação!

Em contrapartida, Retry adaptativo (Retry Token Bucket) ou Retry Circuit-Breaker (o breaker é a nivel de retry, e não complemento a ele) funcionam para para sobrecargas ou indisponibilidades longas do sistema, e também para curtas - embora com menor taxa de sucesso para sobrecarga curta comparada ao backoff+jitter. Ambas as estratégias, em caso de sobrecarga longa, conseguem diminuir BASTANTE a carga da aplicação, para um percentual baixo da carga original, permitindo a aplicação se recupear mais rapido, que é justamente o que se quer em casos de indisponibilidade.

Outro ponto, é que Retry+backoff+jitter funciona muito bem para mitigação (diminuição ou eliminação) da sobrecarga do sistema em cenários mais estáveis (geralmente closed system), ou seja, cenários com long-lived clients ou número de clients limitados e/ou com execução serial/sequencial das requisições, como por exemplo, jobs em background fazendo polling no sistema ou numa fila. Enquanto as estratégias de Retry Token Bucket e Retry Circuit-Breaker, são ideais para cenários onde não há controle no número de clients (unbounded clients), por exemplo, bordas do sistema onde não se tem controle dos usuários ou dos clients - aqui, o importante é estar ciente que nesse tipo de cenário (geralmente open system), sempre haverá novos clients enviando novas requisições ("first try" - o primeiro request) independente se já existem outros usuários (ou threads) fazendo backoff nesse meio tempo.

O autor conseguiu combinar muito bem os vários artigos de resiliência do Marc Brooker e usar o simulador dele para validar as hipoteses! Ficou simplesmente ANIMAL!

(Eu acompanho o Marc, mas confesso que tive que reler os artigos do Marc para relembrar e conectar melhor os pontos - e gesuis, eh animal demais!)

• ⭐️ Google SRE Book: Handling Overload

  • In a majority of cases (although certainly not in all), we've found that simply using CPU consumption as the signal for provisioning works well, for the following reasons:

    • In platforms with garbage collection, memory pressure naturally translates into increased CPU consumption.
    • In other platforms, it's possible to provision the remaining resources in such a way that they're very unlikely to run out before CPU runs out.

  • Our larger services tend to be deep stacks of systems, which may in turn have dependencies on each other. In this architecture, requests should only be retried at the layer immediately above the layer that is rejecting them. When we decide that a given request can't be served and shouldn't be retried, we use an "overloaded; don't retry" error and thus avoid a combinatorial retry explosion.

• ⭐️ Google SRE Book: Addressing Cascading Failures

  • A cascading failure is a failure that grows over time as a result of positive feedback.

  • Limit retries per request. Don’t retry a given request indefinitely.

  • Consider having a server-wide retry budget. For example, only allow 60 retries per minute in a process, and if the retry budget is exceeded, don’t retry; just fail the request. [...]

  • Think about the service holistically and decide if you really need to perform retries at a given level. In particular, avoid amplifying retries by issuing retries at multiple levels: [...]

  • Use clear response codes and consider how different failure modes should be handled. For example, separate retriable and nonretriable error conditions. Don’t retry permanent errors or malformed requests in a client, because neither will ever succeed. Return a specific status when overloaded so that clients and other layers back off and do not retry.

  • If handling a request is performed over multiple stages (e.g., there are a few callbacks and RPC calls), the server should check the deadline left at each stage before attempting to perform any more work on the request. For example, if a request is split into parsing, backend request, and processing stages, it may make sense to check that there is enough time left to handle the request before each stage.

A tal da "Fila virtual"

• ⭐️ SeatGeek Case: Build a Virtual Waiting Room with Amazon DynamoDB and AWS Lambda at SeatGeek
• Youtube: A POLÊMICA "Fila Virtual" da Caixa na Mega-Sena da Virada! O que poderia melhorar? - By André Secco

Queuing, Backpressure, Single Writer and other useful patterns for managing concurrency

Youtube | ScyllaDB: Resilient Design Using Queue Theory: This talk discusses backpressure, load shedding, and how to optimize latency and throughput.

⭐️ SREcon24 Americas - System Performance and Queuing Theory - Concepts and Application

The #1 rule of scalable systems is to avoid congestion collapse - by @jamesacowling
https://x.com/jamesacowling/status/1934991944234770461

A good metaphor for congestion collapse is to imagine you're a barista at a coffee shop that just got popular. The cashier keeps taking orders and stacking them up higher and higher but you can't make coffees any faster. [...] - by @jamesacowling
https://x.com/jamesacowling/status/1935812480254787819

Linkedin post: When your system is overloaded, you have 2 options...

RabbitMQ: Stack Overflow Behavior

RabbitMQ's "stack overflow behavior" refers to how the system handles situations where a queue's capacity is exceeded, leading to an "overflow" of messages. This is particularly relevant when the rate of messages being published to a queue is significantly higher than the rate at which consumers can process them.

---

Key Aspects of RabbitMQ Queue Overflow Behavior

📏 Maximum Queue Length

RabbitMQ allows configuring a max-length for queues. This limit dictates the maximum number of messages a queue can hold. Setting this limit is crucial for preventing queues from consuming excessive memory and resources.

---

⚙️ Overflow Strategies

When a queue reaches its max-length, RabbitMQ employs a defined overflow strategy to determine how new messages are handled:

• drop-head (Default)
  This strategy, the default for classic queues, discards the oldest message in the queue when a new message arrives and the queue is full. This ensures new messages are accepted while maintaining the queue's length limit.

• reject-publish
  With this strategy, newly published messages are rejected (nacked) by the broker when the queue is full. This signals to the publisher that the message could not be enqueued.

• reject-publish-dlx
  Similar to reject-publish, but rejected messages are routed to a Dead Letter Exchange (DLX) if configured. This allows for specific handling of rejected messages, such as logging or retrying.

---

🚦 Flow Control

Beyond explicit overflow strategies, RabbitMQ implements a credit flow control mechanism. This system monitors the rate of message consumption and can temporarily block publishers if consumers are falling significantly behind, preventing the broker from becoming overwhelmed and potentially crashing due to memory exhaustion.

Note: This is distinct from queue overflow but contributes to overall system stability.

---

⏱️ Message TTL (Time-To-Live)

While not an overflow strategy in itself, setting a message-ttl can help manage queue size by automatically expiring messages after a specified time, regardless of whether they have been consumed. This can prevent queues from growing indefinitely due to slow consumers or unconsumed messages.

---

⚠️ Consequences of Overflow

• Message Loss: If drop-head is used, older messages are discarded.
• Publisher Blocking/Errors: If reject-publish or flow control is active, publishers may experience delays or receive errors when attempting to send messages.
• Resource Consumption: Without proper limits and strategies, an overflowing queue can consume excessive memory on the RabbitMQ server, potentially impacting performance or leading to crashes.

---

✅ Addressing Queue Overflow

• Increase Consumer Capacity
  Scale up the number of consumers or optimize consumer processing logic to handle messages more efficiently.

• Optimize Queue Length
  Adjust the max-length based on system requirements and expected message volume.

• Implement Overflow Strategies
  Choose the appropriate x-overflow strategy (for classic queues) based on desired message handling behavior.

• Utilize Message TTL
  Set TTLs for messages or queues to prevent indefinite message accumulation.

• Monitor and Alert
  Implement monitoring to detect queue overflow conditions and trigger alerts for proactive intervention.

Twitter thread about Cascading Failure and Backpressure Fundamentals

I liked this thread because it talks about 2 (two) useful patterns on the consumer side to handle backpressuring.

However, it didn't discuss system overloading in general. But, on another tweet, he gives an excellent perspective on the (main) trade-off that drives the choice between back-pressure (blocking on input) and load shedding (dropping data on the floor):

When a service is overwhelmed, we have exactly two choices: preserve data or preserve latency.

We either build backpressure to queue and risk collapse, or we shed load and risk data loss.

Which side of this trade-off does your system live on, and can you defend why?

Your Circuit Breaker is Misconfigured - by Shopify blog

https://x.com/0xlelouch_/status/2012119370433228824?s=46

Twitter: Difference between Rate limiting and Throttling

TLDR -

The difference is clear in practice.

Rate limiting puts a strict cap like 100 requests in a minute. Anything above that gets rejected right away, usually with a 429 response.

Throttling lets the requests through but adds delay when things go over the allowed speed. So the extra calls are not dropped, just processed slower to avoid overload.

Rate limiting enforces hard rules and blocks excess. throttling shapes the flow and keeps processing everything, only slower.

In many systems both get used together depending on the situation.

Kairos blog: Backpressure (It must be translated to english)

Types of backpressure

We must keep in mind that depending on the objective we are pursuing, some backpressure techniques are more effective than others and can even be combined for greater effectiveness.

We can organize all these techniques into three main groups:

1. Control

• 1.1. Rate Limiting
• 1.2. Windowing
• 1.3. Pull-Based
• 1.4. Adaptive or Feedback control

2. Buffer

• 2.1. Queueing
• 2.2. Batching

3. Drop

• 3.1. Shedding
• 3.2. Sampling

Twitter: If producer is faster than consumer and you do nothing, bad things happen. - by Abhishek Singh

Yes, and most people learn it only after breaking production once.

Backpressure is one of those backend concepts that sounds academic until your system melts at 2 AM.

At a very simple level: Producer = thing generating work
Consumer = thing processing work

If producer is faster than consumer and you do nothing, bad things happen.

Queues grow. Memory grows. GC goes crazy. Latency explodes. Eventually the service dies, not because > of traffic, but because of uncontrolled buffering.

Backpressure is just the system saying: “Slow down, I cannot handle more right now.”

The key idea: Flow control must exist, otherwise speed becomes a bug.

Where people usually mess this up:

1. In-memory queues
   People think queues are safety nets. They are not. An unbounded queue is just delayed OOM.

2. Async everything mindset
   Async does not mean infinite throughput. Async without backpressure just hides the problem until > later.

3. Retries without limits
   Retry storms + no backpressure = self DDOS.

How some of the real systems handle it:

Kafka
Backpressure happens naturally via offsets and fetch sizes. Consumers pull, they are not pushed. If a > consumer slows down, lag increases, not memory usage.

Reactive systems (WebFlux, Rx, streams)
Demand driven model. Consumer explicitly says how much it can handle. Producer is forced to respect > that demand.

gRPC
Flow control at HTTP/2 level. If receiver window is full, sender blocks. This is backpressure baked > into the protocol.

Databases
Connection pools are backpressure. When pool is exhausted, requests wait or fail fast. That is > intentional!

Good backpressure strategies:

1. Bounded queues
   Always have limits. If full, block, drop, or reject.

2. Fail fast
   Returning 429 is healthier than crashing later.

3. Load shedding
   Drop low priority work when system is stressed. Survival > completeness.

4. Pull over push
   Consumers should control the pace whenever possible.

5. Observability
   Queue depth, lag, and processing time should be first class metrics. If you cannot see pressure, you > cannot control it.

Backpressure is not just an optimization. It is a correctness requirement for distributed systems.

Most systems do not fail because of high traffic. They fail because traffic arrives faster than it > can be safely processed.

⭐️ Article: System Design - Capacity Planning e a Teoria das Filas - by Matheus Fidelis

Twitter: Backpressure is key for well-behaved infrastructure - by Ben Dicken

Backpressure is key for well-behaved infrastructure.

When databases, message queues, connection poolers, or other systems get overloaded, how do they deal >with the pressure?

Some software responds by allowing their resources to exhaust and end up in a crash / failure state >(OOM, etc). This behavior is simple from an implementation perspective, but causes cascading issues >that can lead to total system failure and even data loss. Not good!

A better approach: each component applies backpressure to its clients. Backpressure is the idea that, >when a software service detects it is at or nearing capacity (max connections, using all available >memory buffers, etc) it communicates this back to clients either via explicit messages or connection >rejections. In other words, it protects its own health at the cost of declined or slow service to >some clients.

When backends are designed to respect backpressure signals, unexpected load spikes cause degraded >performance, rather than taking the whole system offline.

Twitter: my comment on this tweet

Insights: Handling Head of Line Blocking

Your system can be optimized for speed and still feel slow to users. The reason? Head of line blocking! Juniors optimize the tasks, seniors optimize the queue =)

You can represent most computers as being a "processor" that computes tasks from "a queue". For example, a web server has a queue of HTTP requests it must process.

Now imagine 100 tiny requests stuck behind one gigantic request. That's head-of-line blocking.

It's a bit like waiting in line at the grocery's cashier with 1 item when the person in front of you has a fully loaded cart.

This problem is well understood in queueing theory, and the ways to solve it are with 1) systems capabilities (e.g., do you support preempting tasks) and 2) scheduling (can you decide the order in which tasks execute?)

All solutions come with trade off. You can virtually eliminate head of line blocking by enforcing, with preemption, that no task can run for more than X units of time. But this mean that you risk starving the long running tasks.

Maybe that's not tolerable in your system and you must partition processors, dedicating some for long tasks and some for short tasks (the "express checkout" line at the grocery store.)

Or you must design a more complex algorithm to prefer shorter tasks until long tasks have been too much penalized.

Most head of line blocking comes from variance in tasks duration. The first step is to understand the nature of the workload you're dealing with.

What's your favorite head of line blocking nightmare? :)

A few tips on how to handle database overload

Latest episode of Databased dropped with 6 pieces of Counter-Intuitive Systems Wisdom (based on designing exabyte-scale systems etc):

1. You can't test your way to correctness.

Testing stateful systems is harder than writing the code and requires a level of conceptual understanding you generally won't have unless you designed it.

Tests don't accurately model the real world. Data can change in unexpected ways and sometimes your hardware even lies to you.

(fun story in the podcast about the time we had to have a CPU manufacturer fly out to debug a batch of CPUs branching incorrectly)

2. Slow is worse than broken.

If your system takes 5s to respond instead of 10ms you're now handling 500x more concurrent load. You'll likely enter congestion collapse and fail unrecoverably instead of applying > backpressure and allowing upstream clients/systems to back off.

3. Steady state should be worst state.

Don't design your system to be efficient most of the time but perform expensive retries under failure scenarios. That's the worst time to make your system less efficient. Large systems > have to be provisioned for peak load anyway so just take the hit at steady-state so you don't fail unrecoverably when overloaded.

4. Simple is better than sophisticated.

Simple systems scale. Simple systems are easily understood when things break. Simple systems are extensible when requirements change. Simple systems are way harder to design than > sophisticated ones. Amateurs write complex code.

5. Architecture is more important than performance.

Raw performance numbers are irrelevant if a system is architected to have 20 request waterfalls and no caching. Architectural changes can easily have several orders of magnitude difference > in workload. Never take an architectural hit for some small performance wins.

6. Queues are hard and usually bad.

Both in distributed systems and computer networks, long queues are almost always a huge design mistake that will turn a small blip into a large outage. Small amounts of queueing is often > fine but if your system is receiving more load than it can handle all a queue is going to do is increase latency and guarantee all your requests time out. Queueing theory is very subtle > and "slap a queue in front of it" is usually a bad idea.

Lots more nuance to go along with all of these and some positive steps you can take to avoid disaster at scale. Good thing we made a podcast episode about it...

Stability sim is a simple, interactive, simulator that allows you to explore some of the behaviors that cause long outages even in simple distributed systems, and understand the pitfalls of caches, simple retry strategies, round-robin load balancing, and other common patterns.

• ⭐️ Marc Brooker's tweet
• Stability Sim application
• Stability Sim demo | Youtube
• Stability Sim repository | Github

Most System Design Mistakes Hide Between the Boxes -- by Raul Junco

• Queue depth

A queue can absorb spikes, decouple producers from consumers, and make systems more resilient. But a queue can also hide failure. The system can look healthy while the backlog quietly piles up.

If consumers are slow because they are blocked on database writes, locks, connection pools, or downstream rate limits, adding more consumers only increases pressure on the same bottleneck.

Twitter: @Tef_ebooks thinking out loud about complex systems: logistics infrastructure and its relation to computer programming

⭐️ PlanetScale: The feedback loops behind Kubernetes

An operator is a feedback controller. It's the same closed loop that runs a thermostat or keeps your car at a fixed speed on cruise control.

Kubernetes is not only a container runtime. It's not only a YAML processor, or an orchestrator, or whatever word we use that year. For me, the useful way to read Kubernetes is this: Kubernetes is a framework for feedback controllers, plus a consistent store to hold their setpoints and a shared event bus to wake them up.

[...] The core idea is still the same one we started with: write down what you want, look at what exists, make the next change, and repeat. Events wake the loop up, but the current state decides what happens.

[...] Kubernetes didn't invent these ideas; a thermostat had them long before us. [...] Mechanical and electrical engineers figured out how to build stable, long-running systems before us. Software engineering is still catching up, and Kubernetes gives us a practical way to use those ideas in production.

Queues are not just a buffer, but also a scheduler

LinkedIn post by Max Demoulin (DBOS)

People often think of queues as buffers. But a queue is not just a waiting room.
It is a scheduler.

When using a queue, the question is not only: "Will this task run?"

It also is: "Which task should run next?"

That is where these queue primitives become powerful:

1. Priority
   Let urgent work jump ahead of less important work. Not every task has the same business value.

A billing job, a user-facing notification, and a background cleanup should not always compete equally.

2. Partitioned queues
   Separate work by user, tenant, account, or resource. This prevents one noisy partition from starving everyone else.

It also lets you serialize work where ordering matters, without serializing the whole system.

3. Dedup keys
   Prevent the same logical work from being scheduled twice. If a user clicks "export" five times, you do not need five exports :)

Together, these primitives turn a queue into a policy engine:

• Priority decides what work matters most.
• Partitioning decides what work should be isolated.
• Deduplication decides what work is redundant.

FIFO is usually the default, but scheduling is a design choice, and under load, that choice becomes part of your product.

You can learn System Design while waiting for your coffee

Shared resources need pooling, backpressure, or load shedding.

Tweet: Clickhouse Postgres WAL backpressure

Constant work principle

• ⭐️ Resilience beyond the obvious: The principle of constant work

  • If an emergency mode doesn’t run continuously under real conditions, you don’t have a contingency plan; you have an assumption.

  • [...] trading reactive cleverness for constant predictability is one of the architecture decisions that most increases resilience with the least complexity.

• Scaling Applications with Constant Work Pattern
• AWS: Doing Constant Work to Avoid Failures
• AWS: REL04-BP03 Do constant work
• AWS re:Invent 2018: Close Loops & Opening Minds: How to Take Control of Systems, Big & Small ARC337
• AWS re:Invent 2019: Introducing The Amazon Builders’ Library (DOP328) - Static stability (Resilience in the face of impaired dependencies) - Becky Weiss
• REL05-BP01 Implement graceful degradation to transform applicable hard dependencies into soft dependencies
• REL05-BP02 Throttle requests

AWS re:Invent 2019: [REPEAT 1] Amazon's approach to building resilient services (DOP342-R1) - System overload, limit queue size and end-to-end back pressure

PID Loops and the Art of Keeping Systems Stable