FREE AI & machine learning LESSON · AI & machine learning
Separate fit from generalization
Optimization, representation, and error analysis
Training loss reports optimization on a sample, not usefulness in the world.
A model can underfit because its representation or optimization is inadequate, overfit because it captures sample-specific noise, or fail because the train and deployment distributions differ. Learning curves, slice analysis, ablations, and targeted error taxonomies distinguish these causes better than indiscriminately adding parameters or data.
Every improvement proposal should predict which error category will shrink and which evidence will reveal it.
A benchmark can become part of the training process.
Repeatedly selecting models, prompts, or thresholds against one test set leaks information through human decisions even if the rows never enter gradient descent. Preserve a final untouched evaluation, record selection history, and seek external or time-shifted replication before treating small gains as durable.
A test set stops being a test set when it repeatedly steers the design.