FREE COMPUTER SCIENCE STACK LEVEL · 14 OF 20

Networks & distributed systems

How do independent computers coordinate through failure?

The question this layer answers

How do independent computers coordinate through failure?

Many machines, one service. This layer connects Protocols, Load balancers, Queues, Replication, Consensus to the rest of the computing stack.

A timeout is an observation, not a verdict

A client sends a reservation request. The server commits it, but the response disappears. The client waits until its deadline and times out. From that observation alone, it cannot distinguish a lost request, a slow server, a committed operation with a lost response, or several other failures. Treating timeout as proof of nonexecution causes duplicate work.

A retry protocol can attach a client-generated operation identifier. The server associates that identifier with the authenticated caller, the request's meaning, and the recorded result. A repeated identifier with the same meaning returns the recorded outcome; reusing it with different input must be rejected. Retention duration matters: after deduplication records expire, the original guarantee may no longer apply.

Commit the result and its identity together

If the server first performs the reservation and only later writes its deduplication record, a crash between those steps reintroduces ambiguity. For changes inside one database, put the domain change and the operation record in the same transaction. Concurrent requests with the same key also need a uniqueness constraint or equivalent coordination.

An external side effect, such as sending an email, cannot usually join that local transaction. A transactional outbox can record an intent atomically with the reservation; a separate worker delivers it. The worker may retry, so delivery still needs its own duplicate-handling semantics. “Exactly once” is meaningful only after defining what is counted, where it is recorded, and which failure boundary is covered.

Throughput does not remove waiting

Suppose requests arrive at 120 per second while a worker can finish 100 per second. With a persistent, unlimited queue and no dropped requests, the backlog grows by about 20 per second. After a minute it contains roughly 1,200 additional requests. No choice of retry interval can make a permanently overloaded worker keep up.

Backpressure, bounded queues, admission control, and reduced work can prevent uncontrolled buildup. Retries should respect deadlines and use bounded attempts with delay and jitter. Otherwise a recovering service can be flooded by synchronized retries. Measure end-to-end latency and useful completions, not only how many requests were accepted.

Replicas introduce another coordination problem

Copying data across machines improves some failure properties but does not automatically produce one agreed order of updates. A consensus protocol establishes agreement under specific fault and network assumptions. A majority requirement means some partitions cannot make progress; that restriction protects consistency in the chosen design. Replication is not a universal availability switch.

Your design artifact: draw request, commit, response loss, and retry on a timeline. Mark durable state and trust boundaries. In the lab, make the deduplication key part of the operation contract and calculate whether the service can process its offered load.

Free Networks & distributed systems lessons

1. One request crosses many protocols

DNS, IP, transport, TLS, and HTTP

Naming maps a service name to reachable addresses. IP forwards packets across networks on a best-effort basis. A transport such as TCP provides an ordered byte stream with flow and congestion control; QUIC builds secure multiplexed streams over UDP. TLS authenticates peers and protects bytes. HTTP defines application messages and semantics.

TCP can retransmit lost segments and deliver bytes in order. The server may still reject the request, crash after committing it, or send a response the client never receives. Transport reliability protects a stream between endpoints; application correctness needs operation identity and outcome semantics.

2. Failure is partial and ambiguous

Timeouts, retries, backoff, and overload

The request, server work, or response may have been lost or delayed. Retrying can recover transient failure but may duplicate effects and amplify overload. Deadlines propagate a total time budget; exponential backoff with jitter spreads retry traffic; circuit breakers and admission control protect a struggling dependency.

For a stable system, average concurrency is approximately throughput multiplied by average time in system. At 1,000 requests per second and 200 ms average latency, about 200 requests are in flight. If service capacity falls while arrivals continue, the queue grows and latency rises before outright errors appear.

3. Copies must agree enough

Replication, consistency, ordering, and consensus

Copies receive updates at different times and may be partitioned. A consistency model defines which values and orderings clients may observe. Leader-based replication can serialize writes through one authority; quorum systems overlap read and write sets under specific assumptions; consensus lets a changing group agree on an ordered log despite some failures.

A linearizable operation appears to take effect atomically between call and response, respecting real-time order. Achieving it across replicas requires coordination with an authoritative quorum or leader, which adds latency and can reject work during partitions. Eventual consistency permits more availability but moves conflict and staleness into the application contract.

4. Distributed mastery: shape and observe load

Load balancers, caches, queues, and graceful degradation

Load balancers choose healthy destinations and need a policy for uneven work. Caches trade freshness and invalidation complexity for lower latency and origin load. Queues decouple production from consumption and absorb bursts, but backlog is delayed work, not disappeared work. Bulkheads and degradation preserve critical paths when capacity is scarce.

Cache-aside reads check the cache, load from origin on miss, then populate. On update, deleting the cached value avoids serving a known old copy but races and failures still matter. TTL bounds some staleness; versioned keys or event-driven invalidation strengthen particular workflows. “Use a cache” is incomplete without a freshness contract.

Practice lab: Retry without duplicating work

About 30 minutes. Understand API contracts, operation identity, and durable records from Applications and Databases.

A client asks a worker service to create report R using request ID k. The worker durably records the request and result, but the response is lost. The client times out. Separately, one worker pool can finish 20 jobs per second while arrivals stay at 25 jobs per second.

  1. Draw the message timeline and mark what the client and server each know after the timeout.
  2. Design a safe retry and explain what must survive a worker restart. Handle two workers receiving the same ID concurrently.
  3. Estimate queue growth over 60 seconds if the queue starts empty and rates stay constant. Propose a bounded overload policy.
Hint

Retries need stable identity and an atomic claim on that identity. A queue moves work through time; it does not create capacity.

Worked solution

The server knows R was committed; the client only knows no response arrived in time. Retry k and return the recorded result. Persist the identity, request fingerprint, and result; enforce uniqueness atomically so two workers do not both produce the side effect. If the effect occurs in another system, that system must participate in deduplication or a recoverable workflow, one local table cannot make an arbitrary remote effect exactly once. Queue growth is (25 − 20) × 60 = 300 jobs. Bound the queue, reject or defer excess work, apply retry backoff with jitter and a deadline, and monitor queue age as well as length.

  • Treat timeout as uncertainty rather than proof of failure.
  • Make deduplication survive races and restarts.
  • Explain why sustained overload cannot be fixed by an unbounded queue.

Change one assumption

If a retry arrives after its deduplication record expires, specify whether it can run again. Tie record retention to the advertised retry contract.

Reason about messages that arrive late, twice, or never

Draw event histories. Keep transport reliability, application semantics, consensus, and transaction decisions separate.

1. Trace the layers · allow 35 minutes

Forward the packet, not the application record

A router has 0.0.0.0/0→D, 10.0.0.0/8→A, 10.2.0.0/16→B, and 10.2.3.0/24→C. Route 10.2.3.9, 10.2.4.9, 10.8.3.9, and 11.2.3.9. A sender calls write twice with HELLO and WORLD on one TCP connection. May the receiver obtain HELLOW and then ORLD? Design framing for arbitrary binary messages and explain what DNS, TLS, and HTTP each contribute.

Get a starting hint

Routing chooses the longest matching prefix. Stream reads need not align with writes. A length field itself may arrive in pieces.

Compare your reasoning with the worked solution

The next hops are C, B, A, and D respectively. Prefix specificity, rather than table order, selects the route. TCP presents an ordered byte stream, so receiving HELLOW then ORLD is allowed; two writes do not imply two reads. One framing design uses a fixed-width unsigned length followed by exactly that many payload bytes. Accumulate a partial header, validate length against a configured maximum before allocating, then accumulate the body. Preserve leftover bytes for the next frame. Define byte order and what a truncated connection means.

DNS resolves names using its own records and caching rules. TLS authenticates a peer according to the chosen trust policy and protects channel confidentiality and integrity; it does not establish application authorization or correctness. HTTP defines request/response semantics and its own framing rules. None of these guarantees that a timed-out application request had no effect. A router forwards packets without needing the application's message boundaries.

Check your reasoning

  • Apply longest-prefix match to all four addresses.
  • Handle partial headers, partial bodies, and size limits.
  • Assign guarantees to the correct layer.

2. Calculate and interpret · allow 40 minutes

Bandwidth is not a queueing-delay promise

A path has RTT 80 ms and bottleneck bandwidth 100 Mbit/s. Ignore headers and loss. Calculate the bandwidth-delay product and the best throughput allowed by a 256 KiB outstanding-data window. What if the receive window is 128 KiB while the congestion window is 1 MiB? A stable service completes 200 requests/s with mean time in system 0.15 s. Use Little’s law to calculate mean requests in the system. Why does that not predict p99 latency or prove the system is stable?

Get a starting hint

Convert bits to bytes and KiB to 1,024 bytes. TCP is constrained by both flow-control and congestion windows.

Compare your reasoning with the worked solution

The bandwidth-delay product is 100,000,000×0.08=8,000,000 bits, or 1,000,000 bytes. A 256 KiB window permits at most 262,144/0.08=3,276,800 bytes/s, about 26.2144 Mbit/s, under the simplified steady-state model. With receive window 128 KiB and congestion window 1 MiB, the smaller window limits outstanding data: 131,072/0.08=1,638,400 bytes/s, about 13.1072 Mbit/s. These are bounds, not guaranteed achieved rates; loss, application stalls, overhead, and changing RTT can lower throughput.

Little's law gives L=λW=200×0.15=30 mean requests in the system, including service and waiting if W measures both. The identity relates long-run averages in a stable population with consistent boundaries. It does not supply a latency distribution, and two services with the same mean can have very different tails. Applying it to a growing backlog without checking stability hides the problem. Measure arrival and departure rates, queue size over time, and latency percentiles separately.

Check your reasoning

  • Use the smaller window and consistent units.
  • Define the boundary of the queueing system.
  • Do not infer tail latency or stability from a mean identity.

3. Construct an event history · allow 35 minutes

A timestamp order does not prove causation

Processes P and Q begin Lamport clocks at zero. P performs local event a, then sends m at b. Q performs local event c, receives m at d, then sends n at e. P receives n at f. Increment for each event; on receive set clock to max(local,message)+1. Compute all timestamps. Identify a concurrent pair and explain why comparing Lamport values cannot decide causality. Repeat with two-component vector clocks.

Get a starting hint

The message carries the send event's timestamp. For vectors, merge componentwise and then increment the receiver's own component.

Compare your reasoning with the worked solution

Lamport timestamps are a=1, b=2, c=1, d=max(1,2)+1=3, e=4, f=max(2,4)+1=5. Events b and c are concurrent: neither has a path of local-order and message edges to the other, although their scalar timestamps differ. Happened-before implies increasing Lamport timestamps; the converse is false. A process-ID tie-break can create a deterministic total order, but that added order is not evidence of causation.

With component order (P,Q), vectors are a=(1,0), b=(2,0), c=(0,1), d=(2,2), e=(2,3), f=(3,3). b and c are incomparable componentwise, revealing concurrency in this fixed-membership model. Vectors require space proportional to represented participants and careful treatment of membership and identity. Neither clock reports elapsed physical time or bounds how long a message spent in transit.

Check your reasoning

  • Apply receive updates correctly.
  • Give a concurrent pair with unequal scalar clocks.
  • Distinguish causal order from wall-clock time.

4. Find a linearization · allow 35 minutes

Test a history against the promised object

A register starts at zero. Client A writes 1 and receives success. Only after that response, client B starts a read and receives 0. Is this history linearizable? Could it be sequentially consistent if these are the clients’ only operations? Now a read overlaps the write and returns 0: can that be linearizable? Explain how a stale replica can produce the first history and what a read-your-writes guarantee alone would miss.

Get a starting hint

Linearizability preserves real-time order between non-overlapping operations. Sequential consistency preserves each client's program order, not every cross-client real-time edge.

Compare your reasoning with the worked solution

The first history is not linearizable: the completed write must precede the later read, so the register specification requires 1. It can be sequentially consistent under the given one-operation-per-client history by placing B's read before A's write in a global sequential order. That order preserves both clients' trivial program orders but violates real time. If the read overlaps the write, a linearization can place the read first and return zero; overlap allows either relative order consistent with each operation's interval.

A lagging replica can return the old value after another replica acknowledged the write. Read-your-writes normally constrains a client's reads after its own writes, so it need not protect B from missing A's write. Enforcing a stronger global register contract needs an appropriate read/write protocol and failure assumptions. Waiting for a particular replica position may support a session contract without making all clients' operations linearizable.

Check your reasoning

  • Use operation intervals rather than timestamp labels alone.
  • Construct the sequential-consistency order.
  • State the scope of the session guarantee.

5. Safety argument · allow 45 minutes

A quorum is part of the proof, not the whole protocol

Five replicas A–E use a crash-fault consensus protocol. What majority size is required, and how many crashed replicas can be tolerated while making progress? Show why two majorities intersect. A log entry is stored on A,B,C; could D,E simply elect D and overwrite it? Explain which election and log rules are needed in Raft. Why can an old-term entry not be declared committed merely by counting replicas in the same way as a current-term entry? What happens to a 2–3 partition?

Get a starting hint

Intersection only ensures a shared witness exists. The protocol must require that witness to preserve relevant durable evidence.

Compare your reasoning with the worked solution

A majority is three. Any two three-member subsets of five intersect because 3+3>5. Progress can tolerate two crashes provided the remaining three can communicate, elect a leader, and access required durable state. D,E alone cannot form a majority. To recruit a third voter, a Raft candidate must satisfy the log up-to-date test based on last term and then index; voters also persist term and vote information. These rules, the log-matching checks, and the leader's replication rules connect quorum intersection to safety. Merely storing an entry on three arbitrary machines is not a complete proof of commitment.

Raft directly commits an entry by majority counting only when it is from the leader's current term; committing that entry also commits preceding entries. Old-term entries can have problematic histories during leadership changes, so applying the same counting shortcut to them is unsafe. In a 2–3 partition, the three-side can eventually make progress under the protocol's timing assumptions; the two-side cannot commit new commands. Safety does not imply guaranteed liveness during unlimited delay or repeated failures. This model excludes Byzantine replicas.

Check your reasoning

  • Prove majority intersection.
  • Name durable voting and log restrictions.
  • Separate current-term commitment, old entries, safety, and liveness.

6. Design a retry protocol · allow 50 minutes

A timeout after the effect is the hard case

An order service must debit a local ledger and request shipment from another service. The shipment service may act but lose its response. Design a protocol with a transactionally written outbox and a receiver-side idempotency key. Trace crashes before local commit, after local commit before sending, and after shipment before acknowledgment. Explain why deleting deduplication records too early breaks the contract. Contrast this with two-phase commit when the coordinator fails after participants prepare.

Get a starting hint

The local ledger change and outbox row must share one commit. Deduplication and the receiver's durable effect must also share an atomic boundary or an equivalent protocol.

Compare your reasoning with the worked solution

Assign a stable operation ID to the intended shipment. In one local transaction, debit the ledger and insert an outbox row containing that ID and payload. A worker repeatedly sends committed rows. The receiver atomically records the ID, validates that retries use the same payload, performs its durable shipment-state transition, and records the result. It returns the stored result for repeats. Physical shipping still requires a fulfillment process whose own retry behavior is designed; a database row alone cannot make a physical action atomic.

Before local commit there is neither debit nor outbox row. After commit but before sending, the worker can recover the durable row. After the receiver acts but before acknowledgment, the repeated ID retrieves the original result instead of creating a second shipment. Marking the outbox delivered must happen after a trustworthy acknowledgment. Deduplication retention must cover all possible delayed retries or use a protocol that safely retires identities; forgetting too soon makes an old request appear new.

Two-phase commit instead asks prepared participants to await a global commit/abort decision. A prepared participant generally cannot unilaterally guess the decision after coordinator failure without risking atomicity, so classic 2PC can block. An outbox gives eventual delivery and explicit intermediate states, not instantaneous cross-service atomicity.

Check your reasoning

  • Identify both atomic boundaries.
  • Handle the effect-before-acknowledgment crash.
  • State retention and physical-effect limits.

Study the underlying methods

Section project: Deliver a byte stream over a lossy simulated network

6 to 8 hours. Implement a reliable ordered byte transfer on top of an unreliable local event simulator. Input is a byte string, segment size, retransmission timeout, and a deterministic schedule of packet drops, duplication, and delays. Model data packets with sequence offsets and payloads, and acknowledgements with the next contiguous byte offset expected by the receiver. Begin with stop-and-wait to establish correctness, then allow a small fixed window if time permits. The receiver must buffer or discard out-of-order data according to a stated rule, suppress duplicates, and emit each byte exactly once in order. Both data packets and acknowledgements can be lost. Output a packet-event trace, delivered bytes, retransmission count, and completion or budget-exhausted status. Use a discrete-event priority queue and standard-library Python; do not send packets to public hosts. State that the experiment assumes corruption-free delivered packets and does not implement congestion control, cryptographic protection, or the complete TCP specification. Explain why bounded loss in your test schedule permits progress, while arbitrary permanent loss makes unconditional delivery impossible. Include an explicit end marker or declared byte length so silence cannot be mistaken for completion. Specify how the sender knows that the receiver has accepted the final data or completion marker.

Milestones

  1. Define packet and acknowledgement semantics and event ordering for equal timestamps.
  2. Build the deterministic network simulator and a stop-and-wait sender and receiver.
  3. Add timeout retransmission, duplicate handling, and explicit transfer completion.
  4. Test lost data, lost acknowledgements, duplicates, and reordered events.
  5. Measure retransmission overhead and explain the assumptions required for eventual completion.

Acceptance criteria

  • With no loss, the delivered byte string exactly matches the input, including empty input and a final short segment.
  • Dropping the first data packet still permits completion after a timeout under the bounded-loss schedule.
  • Dropping an acknowledgement causes retransmission without duplicate bytes in the delivered output.
  • Duplicated and delayed packets never move the receiver’s contiguous acknowledgement backward.
  • A schedule dropping every data packet ends as budget-exhausted, not as successful delivery or a mathematical nontermination proof.
  • Repeating a schedule produces the same trace, counts, and final result.

Add a fixed sliding window and compare throughput with stop-and-wait over a simulated high-latency link, then explain why a fixed window is not congestion control.

Practise Networks & distributed systems free →