Generalization and leakage

Prerequisites: Training loss, classifier scores, and confusion matrices.

• Choose a split that matches the deployment question.
• Fit preprocessing only on the training partition.
• Distinguish model selection from final evaluation.

State the claim before splitting the data

Training asks which parameters reduce a chosen objective on observed examples. Evaluation asks how a fitted procedure performs on examples outside that fitting process. A useful experiment begins with a precise prediction time, target, population, and decision. “Predict escalation at the moment a support ticket opens, for next month’s tickets from existing customers” is a testable claim. “The model understands support” is not.

A training set fits parameters. Validation data supports choices such as model family, regularization, stopping point, and threshold. A test set estimates the performance of the selected procedure. Repeatedly adjusting the procedure in response to test results uses that test for selection. Cross-validation repeats training and validation across folds; a final untouched test can remain separate. See scikit-learn’s cross-validation guide.

Draw the information boundary

Consider four ticket columns: opening text, plan at opening, number of previous tickets, and final resolution time. The first three can be legitimate if constructed using only information available at opening. Final resolution time cannot be a feature for that prediction. Its existence in a historical database does not mean it was knowable at the time of the decision.

Leakage also appears in preprocessing. A mean, imputation value, vocabulary, feature selection rule, or principal component learned from all rows can carry held-out information into training. Split first. Fit transformations on the training partition and apply the fitted transformation to validation and test partitions. A pipeline helps enforce this order inside cross-validation. scikit-learn’s common-pitfalls guide demonstrates the failure and correction.

Work a numeric example. Training feature values are [0,2], and a test feature value is 100. Training-only centering learns mean 1, yielding centered training values [−1,1] and test value 99. Centering the full dataset learns mean 34, yielding [−34,−32] and 66. No test label was used, yet the representation used during fitting depends on the test sample. Whether the resulting score rises or falls is secondary: it estimates a different procedure from the one claimed.

Independence comes from the task

If one customer has 200 near-duplicate tickets, randomly scattering those rows between training and test can measure recognition of that customer’s recurring pattern. That may be relevant to future tickets from known customers, but it is weak evidence about new customers. Keep customer groups disjoint when the latter is the target. For forecasting, preserve time order and ensure labels would have arrived before each training cutoff.

Imagine training through March, validating on April, and testing on May. A ticket opened March 31 but resolved April 10 cannot contribute an escalation label to a model supposedly fitted April 1 if the outcome was not yet known. A timestamp split on the opening date alone misses that issue. Label availability is another time boundary.

There is no single universally correct split ratio. Choose a split that resembles the intended use, leaves enough independent units and positive examples for evaluation, and makes the information boundary auditable. Record both row counts and group counts.

Overfitting is an empirical relationship

A flexible model may drive training error down while held-out error rises. Regularization penalizes some kinds of complexity; early stopping selects a training point using validation evidence. Both are choices whose settings need validation. A smaller training loss is neither proof of a better model nor evidence that the evaluation procedure is valid.

Report absolute counts. If a model detects 8 of 10 positives, recall is 80%, but one case changes it by ten percentage points. Detecting 800 of 1,000 positives gives the same point estimate with different sampling uncertainty. These counts still do not reveal every source of uncertainty: correlated observations and future distribution changes can matter more than an idealized random-sample interval.

Inspect errors in prespecified, meaningful slices, such as language or ticket category, while retaining the overall counts. A slice with two examples cannot support a strong comparative claim. Do not keep searching for a favorable slice and present it as if it were planned in advance.

A report another person can audit

Record the data origin and exclusions, prediction time, features and their availability, split rule, baseline, fitting procedure, chosen threshold, raw confusion counts, metric definitions, and limitations. Keep code and split identifiers with the report. Reproducibility requires reconstructing what was done, not merely obtaining a similar headline score.

Use the exercise to redesign a flawed ticket experiment. Explain each change in terms of the question it makes the evaluation answer.

Practice exercise

You have 8,000 support tickets from 400 customers, collected January through June. Predict escalation using information available when a ticket opens. An initial experiment randomly splits tickets, includes final resolution time, standardizes all rows before splitting, and chooses a threshold using test labels. Audit all four decisions. Design one experiment for future tickets from existing customers and a separate one for entirely new customers.

Compare with the worked solution

Final resolution time is unavailable at opening and must be removed. Fit the scaler inside each training fold and reuse its fitted parameters on validation/test data. Select the threshold with validation data. Keep the test set untouched until selection ends. For future tickets from existing customers, use an ordered time split and train only on labels actually available by each cutoff; exclude future-derived features. For new customers, also keep customer groups disjoint. If the claim is about new customers in the future, enforce both time and group constraints. Report how many customers and positive cases remain in each split. Random splitting is not automatically wrong; it is wrong when it measures a different generalization task.

Check your understanding

Which feature is invalid for prediction at ticket opening?

• The final resolution time.
• The opening hour.
• The initial text.
• The customer plan at opening.

Check the reasoning

Final resolution time is only observed later. A historical column can exist in the dataset while still being unavailable at prediction time.

Where should a scaler learn its mean and variance during cross-validation?

• From the full dataset before splitting.
• From the test set.
• From the training portion of each fold.
• From whichever split gives the highest score.

Check the reasoning

Treat fitted transformations as model parameters. Validation and test rows receive transform, not fit.

Learn AI curriculum · Next: Neural networks and backpropagation