Distributed transactions and retry-safe effects

Prerequisites: Transactions, recovery logs, replication, and HTTP semantics.

Learning objectives

• Trace prepare and decision phases of two-phase commit.
• Explain why prepared participants can block.
• Design atomic deduplication and an outbox handoff.

A local transaction cannot automatically make updates on several independent services atomic. Suppose an order database records a purchase while an inventory database reserves stock. If one commits and the other fails, a plain sequence of local commits leaves a partial result. A distributed transaction protocol or a deliberately different application workflow must define how that state is prevented or repaired.

Two-phase commit coordinates a single commit-or-abort decision across participants. In the prepare phase, the coordinator asks each participant whether it can commit. A participant voting yes must durably record enough state to honor the eventual decision and usually retain locks or equivalent reservations. If all required participants vote yes, the coordinator durably records commit and sends the decision. Otherwise it decides abort. Participants record and apply the decision, then release resources according to the protocol.

Trace inventory and orders both voting yes, followed by the coordinator crashing before either learns the decision. A prepared participant cannot safely choose abort merely because a timeout expires: the coordinator may have durably committed and told another participant. It must recover the decision through the protocol. This is the classic blocking risk of two-phase commit. Replicating the coordinator can improve fault tolerance, but does not make atomicity or availability concerns disappear under every failure.

External effects create another boundary. A database transaction cannot roll back an email already delivered or an action performed by a service outside its protocol. A saga breaks work into local transactions with compensating actions, but compensation is a new business action, not time reversal. Refunding a payment does not erase the fact that the original payment occurred. Intermediate states and compensation failures need explicit handling.

Idempotency controls repeated requests. Give a logical operation a stable key scoped to the relevant account and action. In one atomic transaction, enforce key uniqueness, validate that repeated keys carry the same payload, perform the database effect, and store its result. A retry returns that result instead of repeating the effect. If key storage and the effect commit separately, a crash between them reintroduces duplication or lost work. Retention limits also define how long the guarantee lasts.

An outbox couples a local state change with an event record in the same database transaction. A relay later delivers the event. A crash after delivery but before marking it sent can cause redelivery, so consumers still need deduplication or naturally idempotent operations. The outbox guarantees durable intent to deliver within its assumptions, not magical exactly-once processing across all systems.

Use a failure table listing every durable write and every outgoing message. Insert a crash after each step and ask whether recovery can determine the result. Clear operation identities and atomic boundaries are more useful than an unqualified exactly-once claim.

Practice problem

A worker records a request key, commits, then separately increments a balance. Show a crash that causes a lost effect. Reverse the order and show a crash that causes duplication. Propose an atomic database design that fixes both. Then explain what happens if an outbox relay sends an event successfully but crashes before recording delivery, and why a prepared two-phase-commit participant cannot always abort safely on timeout.

Worked solution

If the key commits first and the worker crashes before incrementing, a retry sees the key and incorrectly skips the missing effect. If the increment commits first and the worker crashes before storing the key, a retry increments again. Store the key, validated payload identity, balance update, and result in one transaction protected by a unique key constraint; the duplicate-request path retrieves the committed result. An outbox relay crash after sending causes a later retry to redeliver, so the consumer must handle the event identifier atomically with its own effect. A prepared participant may be unaware of an already durable commit decision, so unilateral timeout abort could disagree with another participant that committed.

Check your understanding

A service commits an idempotency key, then separately commits the effect. It crashes between them. What failure can occur on retry?

• The retry always repairs the gap without checking state.
• The effect necessarily happens twice.
• The key can cause the missing effect to be skipped.
• The database automatically joins the two commits into one transaction.

Explanation

The deduplication record claims completion before the effect exists. Atomic key/effect/result storage avoids that split state.

A participant durably prepared and voted yes, then loses the coordinator before hearing the decision. Why is timeout alone insufficient to abort?

• Every prepared transaction is already visible and committed.
• Timeout proves every participant aborted.
• A yes vote proves the coordinator has not written any log.
• Another participant may already know a durable commit decision.

Explanation

Unilateral abort can disagree with a commit decision already made and delivered elsewhere. The participant needs the recovery protocol’s decision.

Primary reading

• PREPARE TRANSACTION and two-phase commit: PostgreSQL Global Development Group
• Distributed OLTP databases: Carnegie Mellon University
• HTTP Semantics, RFC9110: IETF / RFC Editor

Open the study and save your work →