FREE LESSON · AI & machine learning · 2 OF 4
Gradient Descent, Loss and Regularization
Training is guided search — Models, loss, gradients, and regularization
A model family defines possible functions; training chooses parameters.
Examples flow forward to predictions. A loss turns prediction error into an optimization signal. Gradients estimate how parameter changes affect that loss; an optimizer updates parameters across batches. Regularization, architecture, and data augmentation constrain the search toward functions likely to generalize rather than memorize.
The model optimizes the signal you wrote, not the outcome you meant.
Underfitting and overfitting leave different traces
High training and validation error suggest the model, features, or optimization cannot capture enough signal. Low training error with worsening validation error suggests memorization or distribution mismatch. More capacity helps the first case and can worsen the second; regularization or better data may help the second.
Compare learning curves and error categories before changing the architecture.