Learning Representations by Back-Propagating Errors

Credit assignment is a derivative calculation

The paper explains how hidden units can learn by propagating derivatives of an output error backward through a multilayer network. The backward quantities are sensitivities, not corrected labels sent into earlier layers. This distinction lets you derive an update from the computation itself instead of memorizing an isolated formula. The following tiny network is an original worked example, not one of the paper’s experiments.

Original reconstruction: one hidden unit and one output

Use a scalar input x=2, hidden activation h=σ(wx+b), output z=vh+c, and loss E=(z−t)^2/2. Here σ(s)=1/(1+exp(−s)), and the target is t=1. Choose w=0, b=0, v=2, and c=0.5. The output is deliberately linear, while the hidden unit is sigmoid; this keeps the calculation short and makes the architecture assumption explicit.

The forward pass gives hidden input 0, so h=0.5. Output z is 2×0.5+0.5=1.5. Error z−t is 0.5 and loss is 0.125. Save x, h, z, and the parameters used for this pass, because the derivatives must describe the same computation.

Begin at the loss: ∂E/∂z=z−t=0.5. The output parameter derivatives are ∂E/∂v=(∂E/∂z)h=0.25 and ∂E/∂c=0.5. Sensitivity at the hidden activation is ∂E/∂h=(∂E/∂z)v=1. The sigmoid derivative at h=0.5 is h(1−h)=0.25, so sensitivity to the hidden preactivation is 0.25. Multiplying by the input gives ∂E/∂w=0.5, while ∂E/∂b=0.25.

With learning rate 0.1, one simultaneous gradient step gives w=−0.05, b=−0.025, v=1.975, and c=0.45. “Simultaneous” means compute every gradient using the original forward-pass parameters before replacing them. Updating v first and using its new value in the hidden derivative mixes two computations and no longer follows the derived gradient.

Check the mechanism independently

A centered finite difference approximates the w derivative by [E(w+ε)−E(w−ε)]/(2ε), holding every other parameter fixed. For ε=10^−5 at the original point, this should be close to 0.5. Try several ε values: too large introduces approximation error; too small can magnify floating-point cancellation. Agreement is useful evidence that the implementation differentiates this objective, not proof that the objective measures the desired application behavior.

If an activation feeds two downstream paths, its total derivative is the sum of contributions from both. Forgetting one path is a common implementation error. This is why reverse traversal of a computation graph accumulates sensitivities rather than simply overwriting them whenever a consumer is visited.

What learning does and does not establish

A sigmoid near zero or one has a small derivative, so a large output discrepancy can still yield a small early-layer update. More generally, optimization depends on initialization, scaling, architecture, and step size. A correct gradient does not guarantee a useful minimum, good generalization, or interpretable hidden units.

Keep the historical and mathematical claims separate. The paper helped establish a practical representation-learning method with important precursors; it did not invent the chain rule. A strong replication reports the exact loss, derivative checks, training behavior, and held-out results. Inspect hidden activations as evidence about the trained computation while avoiding the claim that a unit necessarily represents the concept suggested by its most appealing examples.

For a batch, specify whether the loss sums or averages the examples. Averaging divides the accumulated gradient by the batch size. Changing this convention without adjusting the learning rate changes the update magnitude even when each individual derivative remains correct.

import math
def loss(w, b):
    h = 1 / (1 + math.exp(-(2*w + b)))
    return ((2*h)**2 + (-h - 1)**2) / 2
eps = 1e-6
dw = (loss(eps,0)-loss(-eps,0))/(2*eps)
db = (loss(0,eps)-loss(0,-eps))/(2*eps)
assert abs(dw - 1.75) < 1e-7
assert abs(db - 0.875) < 1e-7

• Learning Representations by Back-Propagating Errors: University of Toronto, author-hosted paper