Derivatives, gradients, and the chain rule

Prerequisites: Functions, algebra, vectors, and matrix multiplication.

Learning objectives

• Derive a simple derivative from its limit definition.
• Apply the chain rule to a composed loss.
• Distinguish a descent direction from a guaranteed finite-step improvement.

Local change is a limit

For a real function f, the derivative at x is the limit of (f(x+h)-f(x))/h as nonzero h approaches zero, when that limit exists. It describes local sensitivity, not the average change over an arbitrary interval. For f(x)=x², the quotient simplifies to 2x+h, so f′(x)=2x. At x=3, a small change δ gives the first-order approximation f(3+δ) ≈ 9+6δ. The omitted term is δ².

Differentiability is an assumption. The function |x| has slopes -1 and 1 on opposite sides of zero, so it has no ordinary derivative there. A program that chooses a convention at such a point must state it; the convention does not make the mathematical function differentiable.

Follow a composed computation

Let z=wx+b and L=(z-y)², where x and y are fixed observations. The chain rule gives ∂L/∂w = 2(z-y)x and ∂L/∂b = 2(z-y). At x=3, y=5, w=1, b=0, the prediction is 3, the residual is -2, and the derivatives are -12 and -4. The loss is 4.

A gradient step with learning rate 0.01 changes w to 1.12 and b to 0.04. The new prediction is 3.4 and loss is 2.56. The negative gradient points in the direction of steepest local decrease under the Euclidean norm, but a finite step can overshoot. With learning rate 1, the same update gives w=13, b=4, prediction 43, and loss 1444.

For a differentiable scalar function of several coordinates, the gradient collects its partial derivatives. For a vector-valued function, the Jacobian records each output's derivative with respect to each input. Matrix composition in the chain rule depends on the dimensions and conventions. Reverse-mode differentiation efficiently propagates sensitivity from a scalar output through many intermediate values by applying these local rules.

Check the implementation separately

A central finite difference, (f(x+h)-f(x-h))/(2h), can check a derivative numerically at a smooth point. Too large an h introduces approximation error; too small an h can amplify floating-point cancellation. Compare several scales and use tolerances rather than exact equality.

A zero gradient is not a universal certificate of a minimum. It can occur at a maximum or saddle point. Convexity supplies stronger conclusions under its assumptions. State the domain, differentiability conditions, and objective before interpreting an optimizer's output. A library returning a gradient does not establish that the loss represents the problem you intended to solve.

Practice problem

For L=(wx+b-y)² with x=2, y=7, w=2, and b=1, calculate the prediction, loss, and both partial derivatives. Take a gradient step with learning rate 0.05 and recompute the loss. Then explain why the derivative of |x| at zero cannot be obtained by declaring its central finite difference to be zero.

Worked solution

The prediction is 2×2+1=5, residual -2, and loss 4. The gradient is (-8,-4). The update gives w=2.4 and b=1.2, so the new prediction is 6 and loss 1. The lower loss verifies this particular step, not every possible learning rate. For |x|, the symmetric finite difference at zero is (|h|-|-h|)/(2h)=0, but the one-sided difference quotients are -1 and 1. Their disagreement means the derivative does not exist. A symmetric numerical check can therefore conceal nondifferentiability; inspect the mathematical definition and sample one-sided behavior.

Check your understanding

For f(x)=x², what is the exact error in the approximation f(3+δ)≈9+6δ?

• δ
• 6δ
• δ²
• Zero for every δ

Explanation

Expanding (3+δ)² gives 9+6δ+δ².

Why can a negative-gradient update increase a differentiable loss?

• A finite learning rate can move beyond the region where the local approximation predicts descent.
• Gradients only work for integers.
• Every stationary point is a minimum.
• The chain rule changes the objective.

Explanation

A local derivative does not guarantee improvement for an arbitrarily large finite step.

Primary reading

• Single Variable Calculus: MIT OpenCourseWare
• Multivariable Calculus: MIT OpenCourseWare

Open the study and save your work →