FREE Systems LESSON · Systems
Design the failure contract
Safety, liveness, and degraded operation
“Reliable” is incomplete without a failure model.
A system can preserve safety while losing liveness, remain available while serving stale data, or acknowledge a write that survives one failure but not another. State which faults are inside the model, what the user may observe, and which invariant must never be violated. Only then do redundancy, recovery, and failover become meaningful mechanisms rather than comforting nouns.
Every guarantee has an acknowledgement boundary and a set of excluded failures.
Recovery code is production code you rarely observe.
Backups that have never been restored, replicas that fail over only in diagrams, and circuit breakers with untested thresholds are hopes rather than controls. Failure drills should verify detection, authority, state reconstruction, dependency behavior, and the human decision path—not merely whether a process restarts.
The evidence for recoverability is a successful recovery under the stated model.