Regularization, validation, and learning curves

Prerequisites: Loss functions, gradients, and vector norms.

Learning objectives

• Interpret an L2 penalty.
• Use validation without consuming the test set.
• Read learning curves cautiously.

Fit and complexity are separate terms

For linear regression, one objective is the mean squared prediction error plus λ times the squared Euclidean norm of the coefficient vector. Assume λ ≥ 0. The first term rewards fitting observations. The second discourages large coefficients. The parameter λ controls their relative weight. The intercept is often excluded from the penalty. State that convention because it changes the optimization problem.

Regularization changes the estimator. It can introduce bias while reducing sensitivity to a particular sample. It does not repair an incorrect target, mislabeled data, or a feature that leaks the answer. Feature scaling also matters: changing a feature from meters to millimeters changes the coefficient required for the same predictions and therefore changes its penalty unless preprocessing is adjusted.

A one-dimensional calculation

Consider one coefficient w, one observation x=1, y=2, and objective (w-2)² + λw². Differentiating gives 2(w-2)+2λw=0, so the minimizing coefficient is w=2/(1+λ). With λ=0 it is 2. With λ=1 it is 1. Shrinkage is a consequence of the stated objective, not an extra operation applied after training.

For a general dataset, fit scaling parameters using training data only. Apply that same transformation to validation and test data. If cross-validation is used, each fold must fit its preprocessing on its own training partition. Otherwise information from a validation fold can influence the representation used to predict it.

Diagnose before adding complexity

Plot training and validation errors as training-set size increases. High training and validation errors may indicate insufficient model flexibility, poor features, noisy labels, or failed optimization. Low training error with substantially worse validation error suggests overfitting or a distribution mismatch. These are diagnostic patterns, not proofs of one cause.

Compare against a simple baseline and inspect examples of errors. Check that optimization actually converged under the intended loss. Adding regularization to a model that never learned because the learning rate was inappropriate treats the wrong problem.

Repeatedly tuning on the test set turns it into another validation set. Keep a final evaluation separate from model selection, document the choices made, and report uncertainty appropriate to the sampling process. For time-dependent data, a random split may answer the wrong question; evaluation on later observations better matches forecasting future performance.

Practice problem

Minimize (w-3)² + 2w² by differentiation. Then design a train/validation/test procedure for choosing λ when features need standardization. Identify exactly where the means and standard deviations are estimated.

Worked solution

The derivative is 2(w-3)+4w=6w-6, so w=1. Split the data before fitting preprocessing. Estimate each feature mean and standard deviation on training observations, transform training and validation using those values, fit candidate models on training, and compare validation loss. After selecting λ, either keep that model or refit the complete pipeline on combined training and validation data using newly estimated preprocessing. Evaluate the chosen final pipeline once on the untouched test data. For cross-validation, preprocessing must be fitted separately within each training fold.

Check your understanding

Increasing λ in the one-coefficient example does what?

• Always improves test error.
• Shrinks the fitted coefficient toward zero.
• Changes the observed label.
• Eliminates sampling uncertainty.

Explanation

The minimizer is 2/(1+λ); test performance still depends on the problem.

Where should a validation fold’s scaling parameters come from?

• The full dataset.
• The validation fold alone.
• The final test set.
• That fold’s training partition.

Explanation

The validation observations must not influence the fitted preprocessing.

Primary reading

• Machine Learning course materials: Stanford CS229

Open the study and save your work →