Neural networks and backpropagation

Prerequisites: Linear models, derivatives, multiplication, and Python loops.

• Explain why nonlinear activations change expressiveness.
• Compute derivatives through a small network.
• Separate backpropagation from an optimizer.

Composition is the mechanism

A dense layer multiplies an input vector by a weight matrix and adds a bias vector. If an input has d entries and the layer has h units, its weight matrix can be represented with shape h×d and its bias with shape h. Applying an elementwise nonlinear function produces the hidden representation.

Without nonlinearities, stacking affine layers still gives an affine function: W2(W1x+b1)+b2=(W2W1)x+(W2*b1+b2). More layers alone do not escape that family. ReLU, defined by max(0,z), changes the function by using different linear behavior on either side of zero. A network learns parameters of a computation; the activation does not assign human meaning to a hidden unit.

Forward computation produces intermediate values and a loss. Backpropagation applies the chain rule in reverse, accumulating derivatives for parameters and intermediate values. An optimizer then uses those gradients for an update. Read Dive into Deep Learning’s computational-graph chapter for the general formulation and its storage requirements.

Work one complete graph

Use a single input and a hidden ReLU:

z = w*x + b
h = max(0, z)
prediction = v*h + c
L = (prediction - y)^2 / 2

At x=3, y=2, w=1, b=−1, v=2, c=0, the forward values are z=2, h=2, prediction=4, and L=2. Keep these intermediate values for the backward pass.

Start at the loss. dL/dprediction=prediction−y=2. Since prediction=vh+c, dL/dv=2h=4 and dL/dc=2. The derivative flowing to h is 2v=4. Because z>0, the ReLU’s local derivative is 1, so dL/dz=4. Finally dL/dw=4x=12 and dL/db=4.

Quantity	Forward value	Loss derivative
prediction	4	2
h	2	4
z	2	4
w	1	12
b	−1	4
v	2	4
c	0	2

With learning rate 0.01, update all four parameters together: w=0.88, b=−1.04, v=1.96, c=−0.02. The new hidden value is 1.6 and prediction is 3.116. The new loss is 1.116²/2=0.622728. A single step has reduced this example’s loss. This says nothing yet about other inputs.

Verify the chain rule with perturbations

def loss(w, b, v, c):
    h = max(0.0, w*3.0+b)
    prediction = v*h+c
    return (prediction-2.0)**2 / 2

parameters = [1.0, -1.0, 2.0, 0.0]
expected = [12.0, 4.0, 4.0, 2.0]
epsilon = 1e-5
for j, analytic in enumerate(expected):
    left, right = parameters.copy(), parameters.copy()
    left[j] -= epsilon
    right[j] += epsilon
    numerical = (loss(*right)-loss(*left))/(2*epsilon)
    assert abs(numerical-analytic) < 1e-5

updated = [p-0.01*g for p, g in zip(parameters, expected)]
assert abs(loss(*updated)-0.622728) < 1e-8

The code checks every parameter independently. In a larger graph, an intermediate value can influence the loss through several paths. Its derivative is the sum of contributions from those paths. Forgetting that sum is a common implementation error.

Boundaries of the toy model

For z<0, the local ReLU derivative is zero, so this example sends no gradient to w or b through that unit. At z=0, ReLU is not differentiable; implementations choose a convention, commonly zero. Avoid the kink when using a central finite-difference check to compare against one derivative convention.

Deep networks multiply and sum many local derivatives. Poor initialization, saturation, or scaling can make gradients unhelpfully small or large. Minibatches, initialization, normalization, and optimizer choices affect training, but none replaces a correct loss and evaluation procedure.

In the exercise, use different values and reproduce the full forward and backward passes. Do not update a parameter halfway through calculating gradients. Backpropagation differentiates one specific forward computation, and all its derivatives must refer to that computation.

Practice exercise

Let x=2, target y=1, z=w*x+b, h=max(0,z), prediction=v*h+c, and loss=(prediction−y)^2/2. Start at w=1, b=0, v=0.5, c=0.5. Calculate the forward values and all four gradients. Take one simultaneous step with rate 0.1 and recalculate the loss.

Compare with the worked solution

Forward: z=2, h=2, prediction=1.5, loss=0.125. dL/dprediction=0.5. Therefore dL/dv=1, dL/dc=0.5, dL/dh=0.25, dL/dz=0.25, dL/dw=0.5, dL/db=0.25. Updated (w,b,v,c)=(0.95,−0.025,0.4,0.45). Now z=h=1.875, prediction=1.2, loss=0.02. All gradients came from the original forward pass. The ReLU derivative was 1 because the original z was positive.

Check your understanding

What is dL/dw at the original parameters?

• 0.25
• 0.5
• 1
• 2

Check the reasoning

Multiply the local derivatives: (prediction−y) × v × 1 × x = 0.5×0.5×1×2=0.5.

What does backpropagation compute?

• The best architecture for every dataset.
• A proof of generalization.
• The next training examples.
• Gradients through a computation graph.

Check the reasoning

An optimizer uses these gradients to update parameters. The backward computation and the update rule are separate mechanisms.

Learn AI curriculum · Next: Attention and language models