Regression, prediction, and confounding

Prerequisites: Means, variance, linear equations, and experimental design.

Learning objectives

• Compute and interpret a simple least-squares fit.
• Diagnose residual patterns and extrapolation risk.
• Explain confounding and why predictive accuracy is not causal identification.

A regression model summarizes a conditional relationship

Simple linear regression predicts a response y from a feature x using a line y_hat=a+b*x. Least squares chooses a and b to minimize the sum of squared residuals, where a residual is observed y minus predicted y. This optimization criterion penalizes large errors more heavily than small ones. It is a modeling choice, not proof that the true relationship is linear.

For points (1,2), (2,3), and (3,5), the mean x is 2 and mean y is 10/3. The slope is the sum of (x-mean_x)(y-mean_y) divided by the sum of (x-mean_x)^2, giving 3/2. The intercept is mean_y minus slope times mean_x, or 1/3. Predictions are 11/6, 10/3, and 29/6. Residuals are 1/6, -1/3, and 1/6, whose sum is zero.

The zero residual sum follows from the fitted intercept for ordinary unweighted least squares; it does not establish a good model. A curved pattern in residuals, unequal spread, influential observations, or dependence over time can reveal violated assumptions or missing structure. A line can fit the observed range reasonably and still give absurd predictions far outside it.

Prediction needs an evaluation boundary

Evaluate predictive performance on data not used to choose the model or its settings. Split according to the deployment question: future observations may require a time-based split, while repeated observations from the same person may require keeping that person within one partition. Otherwise information can leak across the boundary and make performance look stronger than it is.

Compare against a simple baseline and report errors in units that matter. A high correlation does not guarantee small prediction errors, and a small average error can hide systematic failures for important subgroups. The target, loss function, and evaluation distribution should match the intended use.

Association does not identify an intervention

Suppose students who spend more time on lessons earn higher scores. Prior preparation or motivation can influence both study time and score, creating confounding. A fitted slope does not automatically tell us how much scores would change if we forced extra study time. Causal interpretation requires assumptions or a design such as random assignment that addresses alternative explanations. Adding more features to a regression is not a universal cure: controlling for variables caused by treatment or selection can introduce new bias. Draw the assumed causal relationships before deciding what to adjust for.

Practice problem

A platform regresses exam score on the number of optional exercises completed and finds a positive slope. Identify two plausible confounders and explain why the slope is not automatically the causal benefit of assigning more exercises. Propose a randomized study that answers a narrower causal question without forcing participation. Then, for the three points in the reading, compute the prediction at x=4 and explain why the arithmetic alone does not validate extrapolation.

Worked solution

Prior knowledge and motivation can both increase optional exercise completion and exam performance. The observed slope combines these associations with any actual learning effect and possibly other mechanisms. One randomized study could assign eligible learners to receive an additional exercise recommendation or the existing interface, then compare a predeclared later assessment outcome by assignment. This estimates the effect of the recommendation policy, not automatically the effect of completing one extra exercise; participation remains voluntary and compliance needs careful interpretation. The fitted line predicts 1/3+(3/2)*4=19/3, about 6.33. That calculation follows the model but x=4 lies beyond the observed range 1 through 3. Additional evidence about the relationship and target population is needed before treating extrapolated predictions as reliable.

Check your understanding

Why does a positive regression slope between optional practice and scores not establish a causal effect?

• Shared causes such as prior preparation can influence both variables.
• Least squares never produces numbers.
• A positive slope always implies reverse causation.

Explanation

Confounding is one plausible alternative explanation; the fitted association alone does not distinguish causal structures.

Which split best avoids leakage when deployment predicts new users and each existing user has many records?

• Randomly split individual records while allowing the same user in both partitions.
• Keep each user entirely within one partition.
• Evaluate only on the training records.

Explanation

Grouping by user prevents identity-specific information from crossing the evaluation boundary for the stated deployment question.

Primary reading

• Introduction to Probability and Statistics: MIT OpenCourseWare

Open the study and save your work →