Reliability objectives and error budgets

Prerequisites: Fractions, latency, and request/response systems.

Learning objectives

• Define the denominator of a service-level indicator.
• Calculate an error budget and burn rate.
• Explain why averages and server-only metrics can mislead.

Measure the promise a user experiences

A service-level indicator is a quantitative measurement of a behavior, such as the fraction of valid reservation attempts that succeed within a deadline. A service-level objective is a target for that indicator over a defined window. Define which events count, how they are observed, and what is excluded before calculating a percentage. A target without a denominator is not an operational contract.

Assume the indicator is successful valid requests divided by all valid requests over 30 days, with a target of 99.9%. If there are 1,000,000 valid requests, the allowed unsuccessful count is 1,000. If 600 have failed, 400 remain in the budget. This is a request-based objective: it does not imply exactly 43.2 minutes of allowable downtime. The minute calculation belongs to a time-based availability measure with its own sampling and traffic assumptions.

Calculate the rate of consumption

The allowed error fraction is 0.001. An observed error fraction of 0.01 burns the budget at ten times the allowed steady rate. A brief spike and a sustained incident can have the same instantaneous ratio but consume different amounts. Alerting should balance urgency with noisy short windows and the amount of budget at risk. The objective is to detect user harm soon enough to act, not to page on every unusual metric.

Latency requires distributional thinking. Nine requests at 10 ms and one at 1,000 ms have mean latency 109 ms. That average describes none of those individual experiences. A tail percentile or a fraction-below-deadline indicator can expose the slow population. State the percentile method and sampling window when comparing small samples; definitions can differ at boundaries.

Locate failures across the path

A server can return HTTP 200 while the browser fails to render the reservation result. Conversely, a request rejected correctly for invalid input may not belong in an availability-failure denominator. Instrument the promised behavior at an appropriate boundary and reconcile client and server observations. Logging every detail indefinitely is not the solution: operational evidence must respect privacy and storage limits.

Use an error budget as a decision aid. A team with sustained budget exhaustion might prioritize reliability work or restrict risky releases. A team comfortably meeting the objective may accept controlled experiments. The policy should be agreed in advance and connected to user needs. A higher numerical target is not automatically better when its cost displaces more valuable work.

Google's SRE treatment provides the terminology and operational context. The arithmetic here illustrates one particular objective; choose indicators from the actual product contract rather than copying a percentage from another service.

Practice problem

A service has a 99.5% request-success objective and 200,000 eligible requests. There have been 700 failures. Calculate its remaining budget. A recent window has a 2% error rate: calculate burn rate and identify one limitation of applying it to the full month.

Worked solution

Allowed errors are 0.005×200,000=1,000. Remaining budget is 300, or 30% of the allowed failures. Recent burn rate is 0.02/0.005=4. This ratio does not establish how long the budget will last unless future request volume and error behavior are modeled. State the window and sample count; a small or unrepresentative recent sample can be misleading. Check whether eligibility and success definitions match the promised user outcome.

Check your understanding

A 99.9% request-success objective on one million eligible requests allows how many failures?

• 100
• 1,000
• 10,000
• 43.2

Explanation

The allowed fraction is 0.001; multiply by the eligible request count.

What does a tenfold burn rate mean?

• The service has ten users.
• Exactly ten minutes remain.
• The observed error fraction is ten times the allowed fraction.
• Latency is ten times its median.

Explanation

Burn rate compares error fractions; time-to-exhaustion requires assumptions about duration and traffic.

Primary reading

• Service Level Objectives: Google SRE book

Open the study and save your work →