Architecture trade-offs under partial failure

Prerequisites: Transactions, interfaces, and network timeouts.

Learning objectives

• Identify which guarantees cross a deployment boundary.
• Compare local transactions with compensating workflows.
• Record a decision with assumptions and reconsideration triggers.

Start with the operation, not a diagram style

A reservation workflow checks availability, assigns a copy, records a payment authorization, and sends a notification. A modular application can keep several responsibilities in one process while maintaining explicit interfaces. Separate services place those responsibilities behind network and deployment boundaries. Either arrangement can be disciplined or tangled. The useful comparison asks what changes independently, what must remain consistent, and which failures become possible.

Suppose inventory and reservation records live in one transactional database. A single transaction can enforce their relationship. Splitting them across independently administered databases removes that convenient atomic boundary. A synchronous HTTP call does not recreate it: the remote operation may succeed while its reply is lost. A local rollback cannot necessarily reverse the remote effect.

Trace a partial completion

Consider three events: reserve inventory, authorize payment, send confirmation. Inventory succeeds; the payment response times out. There are at least two possible worlds: authorization failed, or authorization succeeded and the response was lost. Blindly retrying can create duplicate effects unless the provider supports deduplication. Immediately cancelling inventory may be reasonable policy, but the payment outcome still needs reconciliation.

A workflow can store explicit states such as inventory-held, payment-pending, confirmed, and cancellation-pending. Each transition has an operation ID and recorded outcome. Compensation is a new operation that aims to restore a business condition, not time travel. A refund may incur a fee; a notification cannot be unsent. Therefore “rollback” and “compensate” are not interchangeable promises.

Compare costs that can be measured

Independent deployment may help teams release on different schedules, isolate faults, or scale an unusually expensive component. It also adds network latency, retries, version skew, tracing, and operational ownership. A shared process reduces some coordination costs but can create contention and a larger deployment blast radius. These are trade-offs under a workload and organization, not timeless rankings of architecture labels.

Write a decision record with the required behavior, alternatives, chosen design, assumptions, and a trigger for reconsideration. For example: keep reservation and inventory in one transaction until independent scaling or ownership has demonstrated enough value to justify a distributed consistency protocol. The trigger should reference evidence such as measured contention or release coupling, not an arbitrary company size.

A performance experiment should reproduce relevant traffic and failure modes. A fast happy-path benchmark says little about a stalled dependency or a retry storm. For each boundary, ask what the caller knows after a timeout and which component owns recovery. Those questions make an architecture review concrete.

Practice problem

Compare two designs for the last-copy reservation invariant: one transactional database versus separate inventory and reservation services. List a success trace and a timeout trace for each. State a decision and one measurable condition that would make you revisit it.

Worked solution

With one database, a conditional inventory update and reservation insertion can commit atomically, with retries keyed to an operation ID. With separate services, inventory may commit while reservation creation times out; recovery needs durable workflow state, idempotent requests, and reconciliation or compensation. A defensible initial choice is one transactional boundary when the invariant dominates and scale is manageable. Reconsider when measured workload or independently owned release needs outweigh the added failure protocol. A network boundary does not automatically improve modularity.

Check your understanding

Why is a compensating action different from rollback?

• It always runs in the same transaction.
• It can erase an external observer’s memory.
• It is a new action whose effects may not perfectly reverse the original.
• It never fails.

Explanation

Compensation has its own failure cases and can only approximate restoration of business state.

Which evidence most directly supports splitting a service?

• A larger architecture diagram.
• Measured independent scaling or ownership needs that outweigh coordination costs.
• The number of classes in the codebase.
• A preference for more repositories.

Explanation

Deployment boundaries should solve demonstrated constraints while accounting for the distributed failure model.

Primary reading

• Software Engineering at Google: Google, open book
• How to Design Programs: Felleisen et al.

Open the study and save your work →