Measure performance as an experiment

Prerequisites: Clock frequency, instruction count, and averages.

Learning objectives

• Relate instruction count, CPI, and frequency.
• Design a reproducible comparison with a clear workload.
• Distinguish latency distributions from throughput averages.

Start with an observable question

Performance is not a single number attached to a program. A question might ask how long one request takes, how many requests finish per second, or how much energy a batch consumes. Define the workload, input size, correctness requirement, hardware, and measurement boundary before comparing implementations. A faster result that omits required work is not an optimization of the same task.

For a simplified CPU model, execution time equals instruction count multiplied by average cycles per instruction divided by clock frequency. Suppose implementation A executes one billion instructions at CPI 1.2 and 3 GHz. Its modeled CPU time is 0.4 seconds. Implementation B executes 800 million instructions at CPI 1.8 on the same clock and takes 0.48 seconds. Fewer instructions did not imply less time because their average execution cost increased.

Actual elapsed time can include waiting for storage, scheduling, network responses, and synchronization. Frequency may vary during the run. The equation remains a useful accounting relation when quantities are defined consistently, but it is not an excuse to ignore costs outside the chosen boundary.

Control what can distort the comparison

Run both versions on the same representative inputs and verify identical required outputs. Record compiler settings, runtime version, worker count, and relevant machine conditions. Separate cold-start behavior from a warmed steady state if both matter. Randomize or alternate run order when temperature or background load could bias a long sequence of trials.

Use repeated measurements to expose variability. Report a distribution or suitable quantiles, not only the single best run. A median describes a typical observation; a high percentile describes a tail. Neither substitutes for sample size, the definition of a request, or a description of the load generator. A tiny sample cannot establish a stable extreme percentile.

Aggregate ratios carefully

Suppose one workload improves from ten seconds to five, a speedup of two, while another worsens from ten to twenty, a speedup of one half. The arithmetic mean of speedups is 1.25, but total elapsed time grows from twenty to twenty-five seconds. For these equally counted tasks, total-time ratio is 0.8. The geometric mean of the two speedup ratios is one and describes a different summary.

Choose the aggregation that answers the real operational question. Profiling can then identify where time is spent, while hardware counters can test a specific hypothesis about misses or stalls. A counter correlation suggests an investigation; a controlled intervention and repeated timing provide stronger evidence of cause.

Primary reading

• Performance Measures (MIT OpenCourseWare, Computation Structures).

Practice problem

Two batch tasks originally take 2 and 18 seconds. A change reduces the first to 1 second and the second to 15 seconds. Compute each speedup and the speedup of running one copy of each task. Explain why the unweighted arithmetic mean of the two speedups answers a different question. Propose one correctness check and one experimental control.

Worked solution

The first task speeds up by 2/1=2. The second speeds up by 18/15=1.2. Running one copy of each takes twenty seconds originally and sixteen seconds afterward, so the batch speedup is 20/16=1.25. The unweighted arithmetic mean is 1.6, which gives equal weight to ratios even though the second task dominates elapsed time. It does not predict the duration of this batch. A correctness check could compare complete output hashes against a trusted implementation on representative inputs, with an appropriate numeric tolerance when the specification permits one. An experimental control could alternate old and new versions on the same machine and input set while recording background load and avoiding concurrent benchmark runs.

Check your understanding

Which change alone proves that elapsed time improved?

• A lower source line count.
• A lower instruction count.
• Repeatedly lower elapsed time for the same verified workload under controlled conditions.
• A larger cache-miss counter.

Explanation

Direct measurement of the defined outcome, with correctness and controls, supports the claim.

Why can a median latency hide a service problem?

• A median includes only the slowest request.
• A substantial slow tail may exist even when the middle observation is fast.
• Medians cannot be computed for times.
• Every request must equal the median.

Explanation

A central quantile does not describe rare but important slow requests.

Primary reading

• Performance Measures: MIT OpenCourseWare, Computation Structures

Open the study and save your work →