THEORY: Distributed Transactions and why you should avoid them (2 Phase Commit , Saga Pattern, TCC, Idempotency etc)

avoid-distributed-transactions.md

Distributed Transactions and why you should avoid them

1. Modern technologies won't support it (RabbitMQ, Kafka, etc.);
2. This is a form of using Inter-Process Communication in a synchronized way and this reduces availability;
3. All participants of the distributed transaction need to be avaiable for a distributed commit, again: reduces availability.

Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them. The best solution is to use the Saga Pattern.

[...]

One of the most well-known patterns for distributed transactions is called Saga. The first paper about it was published back in 1987 and has it been a popular solution since then.

There are a couple of different ways to implement a saga transaction, but the two most popular are:

• Events/Choreography: When there is no central coordination, each service produces and listen to other service’s events and decides if an action should be taken or not;
• Command/Orchestration: when a coordinator service is responsible for centralizing the saga’s decision making and sequencing business logic;

Linkedin post: There are at least 7 ways to achieve idempotency -- by Yves Goeleven

A functional core must be idempotent

Idempotence is the property of certain operations whereby the operation can be applied multiple times > without changing the result.

There are at least 7 ways to achieve it

Natural idempotency

Many operations can be designed in a naturally idempotent way, 'Turn On The Light' is a command which > is idempotent by default, it will have the same effect no matter if the lights were on or off to > begin with.

Keep track of prior decisions

Prior to applying a state change to the system, keep track of the decision that you are going to > apply the state change using event sourcing (Light Turned On), so that any subsequent requests to > make the same state change can be derived from prior recorded decisions and ignored.

Side effect checks

Some times a dangerous approach, but often very useful in the real world, is to check for indirect > side effects of a command to determine if a it needs to be performed or not. When the temperature of > the light is over 65C, there is no need to 'Turn On The Light', you've probably done that already (or > your house may be on fire)

Versioning

A special case of the side effect check, applicable when working directly with state, is to add > versioning information to them (like timestamps or sequence numbers) while at the same time also > adding expected version information to any command that intends to alter the target. Whenever the > version information on the target exceeds the version information in the incoming command, you can > discard the command.

Identify and deduplicate

A fourth option is to identify each command with a unique identifier, and keep track of a list of > recently performed commands (command sourcing). If the identifier is on the list, you can discard it.

Partner state machines

Partner state machines are an approach introduced in the legendary paper 'Life Beyond Distributed > Transactions' by Pat Helland. In a conversation between multiple partners, using messaging, each > partner can maintain a state machine for it's communication with any other partner. The state machine > represents the progression of the relationship between them and avoids conflicts by allowing only for > valid state transitions. By checking the state of the conversation, it is possible to ensure each > command is executed only once.

Accept uncertainty

And finally, there is always the option, at least from a business perspective, to live with some > uncertainty and deal with duplicates in a business sense. Very often there are business processes in > place which can correct these scenarios. For example when you performed a duplicate payment, than you > can dedupe it by issuing a credit nota.

---

I tried to argue with him about the use of transactions as a way to achieve idempotency (comment link):

Twitter: Idempotency Is Not Optional (And Why Most Teams Get It Wrong) - by @devXritesh

LinkedIn post: The outbox pattern exists because application code is not durable. - by Henri Maxime DemoulinHenri Maxime Demoulin (DBOS)

⭐️ How to implement a transactional outbox-like pattern using a DBOS workflow

The outbox pattern exists because application code is not durable.

(Belaboring an obvious point I hope)

The outbox pattern exists to compensate for a fundamental limitation of traditional programs: they can crash and lose the rest of their intent.

Consider a simple workflow: insert a row in a database and send an email. If the process crashes between the two operations, the email may never be sent. The outbox pattern addresses this by persisting the intent explicitly. The program inserts a record into an "outbox" table within the same transaction as the database update.

A separate worker later reads that table and performs the side effect (sending the email). Even if the original process crashes, the intent remains recorded and can eventually be executed.

In other words, the outbox pattern turns control flow into data. The system stores "what still needs to happen" so that another process can continue the work.

Durable workflows approach the same problem differently. Instead of externalizing control flow into an outbox table, the workflow runtime persists the execution state of the program itself. If a process crashes after writing to the database but before sending the email, the workflow engine simply resumes execution from the last durable step. The intent to send the email is already part of the persisted workflow state.

From that perspective, the outbox pattern is a workaround for a missing capability: durable execution of application logic.

p.s.: AI agents need durable workflows as a base capability. Check out DBOS inside Pydantic and LlamaIndex :)

⭐️ Async systems scale your system… and your problems.

You don’t need ordering across the entire system. That would destroy scalability.

You need ordering where state matters.

Rule of thumb: enforce ordering at the level where you store state (device, user, order), not globally.

This is where grouping comes in.

By grouping related messages and processing them sequentially within that group, you preserve the correct order for a specific entity while still allowing parallelism across unrelated > entities.

In practice, this means assigning all events for a device, user, or order to the same processing lane. For Kafka, this is the partition key; for SQS FIFO, this is the message group ID; for > Service Bus, this is the session ID.

Within that lane, events are handled in sequence. Across lanes, the system remains fully parallel.

You’re not eliminating concurrency; you’re shaping it.

Articles written about coordination in distributed systems - by Joseph M. Hellerstein

• What Is Coordination, Really?
• Three Lenses on Coordination
• Algorithms Compute Functions. Systems Make Promises

Coordination is not about slowness or synchronization for its own sake. It is the machinery a system needs when correctness depends on excluding futures that might otherwise still occur.

For now, the takeaway is this: coordination is commitment. Commitment burns futures. And the structure of those commitments—which depend on which, which can happen in parallel, which must be sequenced—is what determines how much coordination costs.

[...] coordination is commitment—a vow to avoid futures with undesirable outcomes.