Sampling, estimation, and confidence intervals

Prerequisites: Random variables, expectation, variance, and square roots.

Learning objectives

• Distinguish a population parameter from an estimator.
• Interpret a frequentist confidence procedure correctly.
• Identify uncertainty that a standard error does not measure.

First define what you want to estimate

A parameter describes a population or data-generating process, such as the mean latency of all requests during a specified week. A statistic is computed from observed data. An estimator is a rule for turning a sample into an estimate of the parameter. The sample mean is one such rule, but whether it answers the intended question depends on how observations were selected.

Sampling only successful requests estimates a different quantity from sampling all requests. Sampling only users who volunteer a rating can overrepresent particular experiences. A larger biased sample can produce a very precise estimate of the wrong target. Before calculating uncertainty, identify the population, unit of observation, inclusion mechanism, missing data, and weighting policy.

Quantify sampling variation under a model

Suppose independent request latencies are modeled as normally distributed with unknown mean mu and known standard deviation 20 milliseconds. A sample of 100 requests has mean 150 milliseconds. The sample mean has standard deviation 20/sqrt(100)=2 milliseconds. A two-sided 95% confidence procedure gives approximately 150 plus or minus 1.96*2, or [146.08,153.92].

Under the stated model, repeatedly sampling and constructing intervals this way would cover the fixed population mean in 95% of repetitions. After observing this particular interval, the frequentist statement does not assign a 95% probability to the fixed parameter being inside it. The probability guarantee belongs to the procedure's repeated-sampling behavior. A Bayesian credible interval answers a different question using a prior and likelihood.

Assumptions determine the interval

Known population variance is uncommon. For normally distributed observations with unknown variance, a Student t interval uses the sample standard deviation and a critical value depending on sample size. Large-sample approximations can work more broadly, but heavily skewed or dependent data require care. Bootstrap methods also depend on whether the resampling scheme reflects the sampling process; resampling individual requests independently may misrepresent clustered users or time bursts.

A confidence interval for a mean is not a prediction interval for an individual future request. Individual values vary more than the estimated mean. It is also not a guarantee about tail latency, missing measurements, deployment shifts, or systematic clock errors.

Report the estimate together with sample size, unit, selection method, and assumptions. If requests are correlated within a session, account for that structure rather than treating every click as a new independent person. Precision should reflect the information collected, not the number of rows conveniently available.

Practice problem

Under the same normal model with known standard deviation 20 milliseconds, compare a sample of 25 requests and a sample of 400 requests, both with observed mean 150. Compute the approximate 95% confidence intervals. Explain why the second is narrower without claiming its sample is automatically more representative. Then describe why a dashboard that drops timed-out requests can underestimate user-visible latency even when its interval is extremely narrow.

Worked solution

For n=25, the standard error is 20/5=4, so the interval is 150 plus or minus 7.84, or [142.16,157.84]. For n=400, the standard error is 20/20=1, giving [148.04,151.96]. Sixteen times as many independent observations reduces the standard error by a factor of four. These calculations assume the same target distribution and sampling model. A large sample does not repair systematic exclusion. Dropping timeouts conditions the sample on successful completion, potentially removing the slowest or failed experiences and changing the target. The narrow interval then describes uncertainty about a selected population, not uncertainty about all users' experience. Report timeout rates and define how censored or failed observations enter the measurement instead of silently discarding them.

Check your understanding

A 95% frequentist confidence procedure means what under its assumptions?

• It covers the fixed parameter in 95% of repeated samples.
• 95% of individual observations lie inside every interval.
• The observed interval always contains the true value.

Explanation

Coverage is a long-run property of the interval-generating procedure, not a guarantee for a particular interval or observation.

Which problem can persist even with a million observations and a tiny standard error?

• Sampling bias from excluding dissatisfied users.
• Random sampling variation must become infinite.
• The sample mean cannot be computed.

Explanation

Increasing size reduces modeled random variation but does not automatically correct systematic selection.

Primary reading

• Introduction to Probability and Statistics: MIT OpenCourseWare

Open the study and save your work →