THEORY: Distributed Transactions and why you should avoid them (2 Phase Commit , Saga Pattern, TCC, Idempotency etc)

avoid-distributed-transactions.md

Distributed Transactions and why you should avoid them

1. Modern technologies won't support it (RabbitMQ, Kafka, etc.);
2. This is a form of using Inter-Process Communication in a synchronized way and this reduces availability;
3. All participants of the distributed transaction need to be avaiable for a distributed commit, again: reduces availability.

Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them. The best solution is to use the Saga Pattern.

[...]

One of the most well-known patterns for distributed transactions is called Saga. The first paper about it was published back in 1987 and has it been a popular solution since then.

There are a couple of different ways to implement a saga transaction, but the two most popular are:

• Events/Choreography: When there is no central coordination, each service produces and listen to other service’s events and decides if an action should be taken or not;
• Command/Orchestration: when a coordinator service is responsible for centralizing the saga’s decision making and sequencing business logic;

Twitter: Idempotency Is Not Optional (And Why Most Teams Get It Wrong) - by @devXritesh

LinkedIn post: The outbox pattern exists because application code is not durable. - by Henri Maxime DemoulinHenri Maxime Demoulin (DBOS)

⭐️ How to implement a transactional outbox-like pattern using a DBOS workflow

The outbox pattern exists because application code is not durable.

(Belaboring an obvious point I hope)

The outbox pattern exists to compensate for a fundamental limitation of traditional programs: they can crash and lose the rest of their intent.

Consider a simple workflow: insert a row in a database and send an email. If the process crashes between the two operations, the email may never be sent. The outbox pattern addresses this by persisting the intent explicitly. The program inserts a record into an "outbox" table within the same transaction as the database update.

A separate worker later reads that table and performs the side effect (sending the email). Even if the original process crashes, the intent remains recorded and can eventually be executed.

In other words, the outbox pattern turns control flow into data. The system stores "what still needs to happen" so that another process can continue the work.

Durable workflows approach the same problem differently. Instead of externalizing control flow into an outbox table, the workflow runtime persists the execution state of the program itself. If a process crashes after writing to the database but before sending the email, the workflow engine simply resumes execution from the last durable step. The intent to send the email is already part of the persisted workflow state.

From that perspective, the outbox pattern is a workaround for a missing capability: durable execution of application logic.

p.s.: AI agents need durable workflows as a base capability. Check out DBOS inside Pydantic and LlamaIndex :)

⭐️ Async systems scale your system… and your problems.

You don’t need ordering across the entire system. That would destroy scalability.

You need ordering where state matters.

Rule of thumb: enforce ordering at the level where you store state (device, user, order), not globally.

This is where grouping comes in.

By grouping related messages and processing them sequentially within that group, you preserve the correct order for a specific entity while still allowing parallelism across unrelated > entities.

In practice, this means assigning all events for a device, user, or order to the same processing lane. For Kafka, this is the partition key; for SQS FIFO, this is the message group ID; for > Service Bus, this is the session ID.

Within that lane, events are handled in sequence. Across lanes, the system remains fully parallel.

You’re not eliminating concurrency; you’re shaping it.

Articles written about coordination in distributed systems - by Joseph M. Hellerstein

• What Is Coordination, Really?
• Three Lenses on Coordination
• Algorithms Compute Functions. Systems Make Promises

Coordination is not about slowness or synchronization for its own sake. It is the machinery a system needs when correctness depends on excluding futures that might otherwise still occur.

For now, the takeaway is this: coordination is commitment. Commitment burns futures. And the structure of those commitments—which depend on which, which can happen in parallel, which must be sequenced—is what determines how much coordination costs.

[...] coordination is commitment—a vow to avoid futures with undesirable outcomes.