Floating-point stability and problem conditioning

Prerequisites: Algebra, vectors, numerical representations, and basic Python.

Learning objectives

• Distinguish absolute and relative error.
• Explain catastrophic cancellation and a stable reformulation.
• Separate a poorly conditioned problem from an unstable algorithm.

Representation is approximate

Floating-point numbers store a finite set of values using a sign, exponent, and significand. Most real numbers cannot be represented exactly, and each arithmetic operation can introduce rounding. Precision controls spacing between nearby representable values; range controls how large or small magnitudes can be. More range does not automatically mean more precision.

Absolute error is the distance between an approximation and its target. Relative error divides that distance by the target's magnitude when the target is nonzero. Near zero, an application often needs an absolute tolerance as well as a relative one. A displayed decimal string can conceal the exact stored approximation, so formatting a result is not the same operation as improving its accuracy.

Equivalent formulas can behave differently

Consider f(x)=sqrt(x+1)-sqrt(x) for large positive x. The two roots are close, and their subtraction can lose significant digits. Rationalizing gives the mathematically equivalent expression 1/(sqrt(x+1)+sqrt(x)). This avoids subtracting nearby large values.

import math

x = 1e16
direct = math.sqrt(x + 1) - math.sqrt(x)
stable = 1 / (math.sqrt(x + 1) + math.sqrt(x))
assert direct == 0.0
assert math.isclose(stable, 5e-9, rel_tol=1e-15)

terms = [1e16, 1.0, -1e16]
naive = 0.0
for term in terms:
    naive += term
assert naive == 0.0
assert math.fsum(terms) == 1.0

These examples assume the usual binary64 Python float implementation. The exact real expression is nonzero, but rounding in the direct calculation erases the small difference. The summation example similarly loses a small term between large magnitudes. A specialized summation method can preserve information that naive accumulation discards.

Conditioning belongs to the question

A problem is ill-conditioned when small relative changes in its inputs can cause large relative changes in its answer. An algorithm is unstable when its own rounding behavior creates unnecessarily large error for the problem being solved. A stable algorithm cannot recover information absent from inaccurate measurements.

For example, subtracting two nearly equal measured quantities can be intrinsically sensitive: tiny measurement errors may dominate the small true difference. In linear systems, an invertible matrix can still be nearly singular, making solutions highly sensitive to perturbations. Exact invertibility is not sufficient evidence of reliable numerical answers.

Diagnose the source of uncertainty before choosing a remedy. Better formulas or compensated arithmetic address algorithmic loss; higher precision may help rounding; improved measurements or reformulating the scientific question may be needed for conditioning. Report units, tolerances, input accuracy, and residual checks instead of treating many printed digits as evidence of correctness.

Practice problem

Two measured lengths are 100.00 and 99.99 units, each with possible absolute error at most 0.02. Compute the reported difference and a worst-case absolute error bound for that difference. Explain why using more floating-point digits cannot resolve the measurement uncertainty. Then compare summing [1e16,1,-1e16] in the displayed order with grouping the large cancellation first. State what a robust test should compare and why exact floating-point equality is sometimes appropriate but often unsuitable for approximate formulas.

Worked solution

The reported difference is 0.01 units. The two errors can have opposite signs, so the difference has absolute error at most 0.04 units, four times the reported magnitude. The true difference may even have the opposite sign. Extra arithmetic precision cannot identify the unknown measurement errors. For the sum, naive left-to-right accumulation rounds away the one before cancellation, producing zero; grouping 1e16 with -1e16 first leaves one. This illustrates nonassociativity of floating-point arithmetic. A robust numerical test compares with a trustworthy reference using tolerances justified by the model, often combining absolute and relative bounds. Exact equality is appropriate for some exactly representable quantities or deliberate representation checks, but approximate formulas generally need an error-based assertion. A tolerance should not be so loose that a numerically broken implementation passes.

Check your understanding

Which distinction is correct?

• Conditioning describes input sensitivity of the problem; stability describes error behavior of an algorithm.
• Conditioning and stability both mean execution speed.
• A stable algorithm removes all measurement error.

Explanation

The problem can be sensitive even when the implementation introduces little additional error.

Why can the rationalized square-root expression be more accurate for large x?

• It avoids subtracting nearly equal rounded roots.
• It makes every real number exactly representable.
• It changes the mathematical function to a constant.

Explanation

The algebraic reformulation reduces cancellation while preserving the intended real-valued expression.

Primary reading

• Floating-Point Arithmetic: Issues and Limitations: Python Software Foundation
• Linear Algebra: MIT OpenCourseWare

Open the study and save your work →