FREE AI & machine learning LESSON · AI & machine learning
Begin with the decision
Targets, proxies, loss, and generalization
The model optimizes the signal you wrote down.
A prediction is useful only inside a decision. Define who acts, what action changes, which errors cost more, and when the truth becomes knowable. The training target is often a proxy for that outcome; the loss compresses its costs; the dataset samples a changing world. A high aggregate score can therefore coexist with a harmful system.
Evaluation is a claim about future decisions, not a report about historical rows.
Leakage is information from a future the model will not have.
Random row splits can put the same customer, device, or episode on both sides. Features recorded after an outcome can silently encode the label. Hyperparameter selection can overfit the validation set. Draw a time-and-entity boundary around the prediction moment, then audit how every feature is produced.
Ask of every feature: could this exact value exist when the decision is made?