Linear models and gradients

Prerequisites: Algebra and Python functions. The chapter introduces derivatives as local rates of change.

• Compute predictions, residuals, and mean squared error.
• Derive both gradients for a line.
• Check an analytic gradient against a finite difference.

Define the object being fitted

A supervised training example pairs input features x with an observed target y. A one-feature linear model predicts y_hat=w*x+b. The weight w and intercept b are parameters learned from data. The feature definition, choice of model, learning rate, and stopping rule are design choices. Prediction applies the fitted parameters to an input; training changes them.

For a vector of d features, the prediction is the dot product w·x+b: multiply corresponding entries and add them. For x=[2,3], w=[0.5,−1], and b=4, the prediction is 1−3+4=2. A feature’s unit matters. Replacing meters by millimeters changes the scale of a corresponding weight and of its gradient.

Loss makes the fitting objective explicit

Define residual r_i=w*x_i+b−y_i for example i. Mean squared error is L=(1/n)Σ r_i². Squaring prevents positive and negative errors from canceling and gives large residuals disproportionate influence. This is an objective choice, not a universal definition of a good prediction. See Google’s linear-regression loss lesson for comparisons with absolute error.

Use observations (0,1), (1,3), and (2,5). At w=1, b=0, the predictions are [0,1,2], residuals are [−1,−2,−3], and MSE is (1+4+9)/3=14/3. At w=2, b=1, all three residuals are zero. This line perfectly fits the toy data. It does not prove that future measurements will have zero noise.

A derivative measures local change

The derivative of loss with respect to one parameter asks how quickly loss changes for a small change in that parameter, keeping the others fixed. Because the derivative of r² is 2r times the derivative of r, the gradients are:

dL/dw = (2/n) sum_i ((w*x_i + b - y_i) * x_i)
dL/db = (2/n) sum_i (w*x_i + b - y_i)
w_new = w - learning_rate * dL/dw
b_new = b - learning_rate * dL/db

Compute both gradients at the same old parameter values. Gradient descent moves against the local gradient. A step that is too large can increase loss or diverge, even for this convex objective. Convexity alone does not make arbitrary learning rates safe. Google’s gradient-descent lesson provides an additional worked derivation.

For our three points at w=1,b=0, dL/dw=(2/3)(0−2−6)=−16/3 and dL/db=(2/3)(−6)=−4. With rate 0.1, the new parameters are w=23/15≈1.5333 and b=0.4. Predictions become approximately [0.4,1.9333,3.4667]. MSE drops from 4.6667 to about 1.2830. Calculate that value yourself before running the code.

def loss(points, w, b):
    return sum((w*x+b-y)**2 for x, y in points) / len(points)

def gradients(points, w, b):
    residuals = [(x, w*x+b-y) for x, y in points]
    n = len(points)
    return (2*sum(x*r for x, r in residuals)/n,
            2*sum(r for _, r in residuals)/n)

points = [(0, 1), (1, 3), (2, 5)]
w, b = 1.0, 0.0
dw, db = gradients(points, w, b)
new_w, new_b = w - 0.1*dw, b - 0.1*db
assert loss(points, new_w, new_b) < loss(points, w, b)

epsilon = 1e-5
numerical = (loss(points, w+epsilon, b)
             - loss(points, w-epsilon, b)) / (2*epsilon)
assert abs(numerical - dw) < 1e-6
print(round(loss(points, new_w, new_b), 4))  # 1.283

Optimization and generalization are separate questions

The finite-difference check compares an analytic derivative with a small numerical perturbation. It is a debugging tool, not a proof that the model suits the task. Extremely small perturbations can magnify floating-point cancellation; extremely large ones stop measuring local behavior.

Full-batch gradient descent uses all training examples for an update. A mini-batch estimates a gradient from a subset, which changes both computational cost and update noise. Later, we will separate fitting from evaluation using held-out data. For now, verify the mechanism: the equation, the residuals, both derivatives, and the simultaneous update.

Practice exercise

For points (1,2), (2,4), use prediction w*x+b and mean squared error. Begin at w=0, b=0. Calculate the loss and both gradients. Update both parameters with learning rate 0.1, then calculate the new loss. Repeat the update once using only the new parameters.

Compare with the worked solution

Initial residuals are −2 and −4, so MSE=10, dL/dw=−10 and dL/db=−6. The first simultaneous update gives w=1, b=0.6. Residuals become −0.4 and −1.4, so MSE=1.06. New gradients are −3.2 and −1.8. The second update gives w=1.32, b=0.78. Predictions are 2.10 and 3.42; MSE=(0.01+0.3364)/2=0.1732. Updating b using a gradient recomputed after changing w would be a different algorithm.

Check your understanding

What is the first updated pair (w, b)?

• (1, 0.6)
• (0.6, 1)
• (−1, −0.6)
• (1, 0.4)

Check the reasoning

Subtract 0.1 times each gradient, evaluating both gradients at the original parameters.

Training loss becomes infinite after increasing the learning rate. What should you investigate first?

• A larger rate always proves the data is wrong.
• Whether the updates overshoot and diverge.
• Whether validation labels are missing from training.
• Whether the model needs more output classes.

Check the reasoning

Convex loss does not make every step size safe. Track loss and parameter magnitudes; use a smaller rate and check feature scale and numerical stability.

Learn AI curriculum · Next: Classification and decisions