THEORY: Distributed Transactions and why you should avoid them (2 Phase Commit , Saga Pattern, TCC, Idempotency etc)

avoid-distributed-transactions.md

Distributed Transactions and why you should avoid them

1. Modern technologies won't support it (RabbitMQ, Kafka, etc.);
2. This is a form of using Inter-Process Communication in a synchronized way and this reduces availability;
3. All participants of the distributed transaction need to be avaiable for a distributed commit, again: reduces availability.

Implementing business transactions that span multiple services is not straightforward. Distributed transactions are best avoided because of the CAP theorem. Moreover, many modern (NoSQL) databases don’t support them. The best solution is to use the Saga Pattern.

[...]

One of the most well-known patterns for distributed transactions is called Saga. The first paper about it was published back in 1987 and has it been a popular solution since then.

There are a couple of different ways to implement a saga transaction, but the two most popular are:

• Events/Choreography: When there is no central coordination, each service produces and listen to other service’s events and decides if an action should be taken or not;
• Command/Orchestration: when a coordinator service is responsible for centralizing the saga’s decision making and sequencing business logic;

⭐️ The Write Last, Read First Rule: Keeping systems in synch

⭐️ Building a Durable Execution Engine With SQLite - by Gunnar Morling

• GitHub repository

On Idempotency Key - by Gunnar Morling
A good write-up on Twitter about the Gunnar's article - by Abhishek Singh (@0xlelouch_)

A good description on the difference between Deduplication and Idempotency

🔗 Post on Linkedin by Henri Maxime DemoulinHenri Maxime Demoulin

Many engineers confuse these two: deduplication keys vs idempotency keys. They look similar but solve two completely different problems.

Deduplication keys are used to detect that a unit of work has already been executed and skip the execution entirely.

Their goal is to prevent the work from running twice.

Idempotency keys are used to ensure that even if a unit of work runs twice (or more), the final state is correct and equivalent.

One prevents execution, the other effects.

Deduplication keys have been around for a while, e.g., SQS uses them to ensure only once delivery within 5 minute windows.

Before you ship your next API or background worker, ask yourself: do you need idempotency, deduplication... or both?

Idempotence comes in different shapes - by Dominik Tornow

Idempotence comes in different shapes

Idempotence is the guarantee that repeating a request yields the same outcome (or, more formally, does not change the state of the system beyond the initial application)

In practice, idempotence comes in a few variants, most notably positive and negative idempotence

Positive

Positive idempotence denotes that the system has accepted the request in the past:

I have accepted this request in the past, I will accept the request again, I will apply this request again, nothing changes

-or-

I have accepted this request in the past, I will accept the request again, I will not apply this request again

Negative

Negative idempotence denotes that the system has rejected the request in the past

I have rejected this request in the past, I will reject this request again

Negative idempotence is often harder to guarantee:

When a system accepts a request, the system state changes, the new state is evidence of the past acceptance of the request

When a system rejects a request, the system state may not change, there is no evidence of the past rejection of the request

⭐️ Real idempotence is about resilience to time and change, not just repetition. - by Abhishek Singh

Linkedin post: Stop using the "Outbox pattern". It's 2026: you can use Durable Execution instead -- by Henri Maxime Demoulin

Stop using the "Outbox pattern". It's 2026: you can use Durable Execution instead.

want to write to system A and B atomically. But I can't write:

• WriteA()
• WriteB()

If the process crash between the two, WriteB() is lost forever.

The outbox pattern says:

• Write to A and an intent to write to B atomically (e.g., in the same transaction or the same document)
• Let another system, a "message relay", eventually send to B.

Eventually, the message relay will pick up on the message and A and B will be in sync.

But consider the downsides:

• You need an external polling system
• You need to move all the compensation logic (for atomicity) in the message relay

i.e., your code becomes a living nightmare and even Claude will struggle to get it right.

With durable execution, you are guaranteed that your program will eventually complete.

So you can finally write:

• WriteA()
• WriteB()

And if the process crash between A and B, the durable execution engine will resume execution where it left of, that is, A, and proceed > to writing to B.

Of course, external systems need to be idempotent, because you get at-least-once execution (like the outbox pattern.)

Who's going to defend the outbox pattern here? :)

---

I liked this comment. Something to think about:

Linkedin post: There are at least 7 ways to achieve idempotency -- by Yves Goeleven

A functional core must be idempotent

Idempotence is the property of certain operations whereby the operation can be applied multiple times > without changing the result.

There are at least 7 ways to achieve it

Natural idempotency

Many operations can be designed in a naturally idempotent way, 'Turn On The Light' is a command which > is idempotent by default, it will have the same effect no matter if the lights were on or off to > begin with.

Keep track of prior decisions

Prior to applying a state change to the system, keep track of the decision that you are going to > apply the state change using event sourcing (Light Turned On), so that any subsequent requests to > make the same state change can be derived from prior recorded decisions and ignored.

Side effect checks

Some times a dangerous approach, but often very useful in the real world, is to check for indirect > side effects of a command to determine if a it needs to be performed or not. When the temperature of > the light is over 65C, there is no need to 'Turn On The Light', you've probably done that already (or > your house may be on fire)

Versioning

A special case of the side effect check, applicable when working directly with state, is to add > versioning information to them (like timestamps or sequence numbers) while at the same time also > adding expected version information to any command that intends to alter the target. Whenever the > version information on the target exceeds the version information in the incoming command, you can > discard the command.

Identify and deduplicate

A fourth option is to identify each command with a unique identifier, and keep track of a list of > recently performed commands (command sourcing). If the identifier is on the list, you can discard it.

Partner state machines

Partner state machines are an approach introduced in the legendary paper 'Life Beyond Distributed > Transactions' by Pat Helland. In a conversation between multiple partners, using messaging, each > partner can maintain a state machine for it's communication with any other partner. The state machine > represents the progression of the relationship between them and avoids conflicts by allowing only for > valid state transitions. By checking the state of the conversation, it is possible to ensure each > command is executed only once.

Accept uncertainty

And finally, there is always the option, at least from a business perspective, to live with some > uncertainty and deal with duplicates in a business sense. Very often there are business processes in > place which can correct these scenarios. For example when you performed a duplicate payment, than you > can dedupe it by issuing a credit nota.

---

I tried to argue with him about the use of transactions as a way to achieve idempotency (comment link):

Twitter: Idempotency Is Not Optional (And Why Most Teams Get It Wrong) - by @devXritesh

LinkedIn post: The outbox pattern exists because application code is not durable. - by Henri Maxime DemoulinHenri Maxime Demoulin (DBOS)

⭐️ How to implement a transactional outbox-like pattern using a DBOS workflow

The outbox pattern exists because application code is not durable.

(Belaboring an obvious point I hope)

The outbox pattern exists to compensate for a fundamental limitation of traditional programs: they can crash and lose the rest of their intent.

Consider a simple workflow: insert a row in a database and send an email. If the process crashes between the two operations, the email may never be sent. The outbox pattern addresses this by persisting the intent explicitly. The program inserts a record into an "outbox" table within the same transaction as the database update.

A separate worker later reads that table and performs the side effect (sending the email). Even if the original process crashes, the intent remains recorded and can eventually be executed.

In other words, the outbox pattern turns control flow into data. The system stores "what still needs to happen" so that another process can continue the work.

Durable workflows approach the same problem differently. Instead of externalizing control flow into an outbox table, the workflow runtime persists the execution state of the program itself. If a process crashes after writing to the database but before sending the email, the workflow engine simply resumes execution from the last durable step. The intent to send the email is already part of the persisted workflow state.

From that perspective, the outbox pattern is a workaround for a missing capability: durable execution of application logic.

p.s.: AI agents need durable workflows as a base capability. Check out DBOS inside Pydantic and LlamaIndex :)

⭐️ Async systems scale your system… and your problems.

You don’t need ordering across the entire system. That would destroy scalability.

You need ordering where state matters.

Rule of thumb: enforce ordering at the level where you store state (device, user, order), not globally.

This is where grouping comes in.

By grouping related messages and processing them sequentially within that group, you preserve the correct order for a specific entity while still allowing parallelism across unrelated > entities.

In practice, this means assigning all events for a device, user, or order to the same processing lane. For Kafka, this is the partition key; for SQS FIFO, this is the message group ID; for > Service Bus, this is the session ID.

Within that lane, events are handled in sequence. Across lanes, the system remains fully parallel.

You’re not eliminating concurrency; you’re shaping it.

Articles written about coordination in distributed systems - by Joseph M. Hellerstein

• What Is Coordination, Really?
• Three Lenses on Coordination
• Algorithms Compute Functions. Systems Make Promises

Coordination is not about slowness or synchronization for its own sake. It is the machinery a system needs when correctness depends on excluding futures that might otherwise still occur.

For now, the takeaway is this: coordination is commitment. Commitment burns futures. And the structure of those commitments—which depend on which, which can happen in parallel, which must be sequenced—is what determines how much coordination costs.

[...] coordination is commitment—a vow to avoid futures with undesirable outcomes.