ImageNet Classification with Deep Convolutional Neural Networks

How to read the original

1. Read The Architecture and distinguish architectural choices from training choices.
2. Read Reducing Overfitting, including Data Augmentation and Dropout.
3. Read the reported results with their dataset, metric, and model variant attached.

A result comes from a coupled system

The paper’s image-classification result combines convolutional structure, substantial labeled data, GPU computation, activation choices, augmentation, and regularization. It is not evidence that one isolated ingredient explains every improvement. Read each comparison with its dataset, evaluation metric, and model variant attached. In particular, top-1 and top-5 error answer different questions, and an ensemble result should not be silently attributed to a single model.

Original reconstruction: count what weight sharing saves

Consider an 8×8 RGB image, represented by 8×8×3=192 scalar inputs. Apply four 3×3 convolutional filters with stride one and no padding. Each filter spans all three input channels, so it contains 3×3×3=27 weights and one bias. Four filters therefore use 4×28=112 learned parameters.

The spatial output width and height are each 8−3+1=6. With four output channels, the layer produces 6×6×4=144 activations. The same filter weights are reused at every spatial position. An edge-like pattern can therefore be tested at multiple locations without learning an independent parameter set for each location.

A fully connected layer mapping the same 192 inputs to 144 outputs has 192×144+144=27,792 parameters including biases. This numerical difference is a consequence of local connectivity and sharing, not proof that the convolutional layer can represent every function the dense layer can. The architectures impose different assumptions about the task. Sharing can help when local visual patterns recur across positions and can constrain the model when position-specific interactions matter.

For one convolutional output, multiply 27 input values by their associated weights, sum them, and add the bias. Sliding the filter changes which input patch is used, not the weights. This trace distinguishes an activation count from a parameter count, two quantities that are often confused when estimating memory or model capacity.

Separate optimization and generalization experiments

A rectified linear activation returns max(0,z). For a positive input its derivative is one; for a negative input it is zero, with the boundary handled by the implementation’s convention. This avoids the same positive-side saturation behavior as a sigmoid, but does not guarantee useful gradients for every unit or every initialization.

Augmentation changes the examples the optimizer encounters. A label-preserving horizontal flip might be reasonable for some object categories but wrong for a task that distinguishes left from right. Choose transformations from the task semantics rather than copying an image-classification recipe without checking labels.

Dropout changes training by randomly suppressing selected activations under a specified scaling convention. Training and inference must use compatible conventions. Do not compare two implementations that differ in both dropout scaling and learning-rate schedule and attribute the whole effect to regularization alone.

Build a measured replication

Use a small public or synthetic dataset with a fixed train, validation, and test split. Compare a baseline, one architectural change, and one data transformation while recording training error, validation error, runtime, and resource use. Repeat runs when randomness changes conclusions. Tuning against the test set weakens the final claim even if the test accuracy becomes impressive.

For a five-class toy example, a correct label ranked fourth is a top-1 error but a top-5 success; with only five classes that metric is trivial. This reveals why a metric must be interpreted in context. The paper’s lasting engineering lesson is to connect inductive bias, optimization, data, and measurement, while keeping the evidence narrower than a general claim of human-like visual understanding.

Reconstruction exercise

For a 10×10 input with two channels, apply six 3×3 filters with stride one and no padding. Compute parameter count including biases and output shape. Compare with a dense layer producing the same number of outputs. Then propose an augmentation that could change the correct label for a direction-recognition task and explain how that affects an ablation.

Worked solution

Each filter has 3×3×2=18 weights plus one bias, so six filters use 114 parameters. Output width and height are 10−3+1=8, giving shape 8×8×6 and 384 activations. A dense layer from 10×10×2=200 inputs to 384 outputs uses 200×384+384=77,184 parameters. These models have different connectivity constraints, so equal output counts do not make them equivalent function classes. A horizontal flip can turn a left-pointing arrow into a right-pointing arrow. Keeping the old label would introduce contradictory supervision for a direction-recognition task. An augmentation ablation should either update the label under a known transformation or omit that transformation, then hold the rest of the training and evaluation procedure fixed. Otherwise a measured change could reflect label corruption rather than a meaningful regularization effect.

Test the limits and transfer the idea

1. Derivation · allow 40 minutes

Account for a convolutional stack

Use this original small model to investigate the paper's convolutional design. A 32 by 32 RGB image passes through two convolutions: first, eight 5 by 5 filters with stride 2 and padding 2; second, sixteen 3 by 3 filters with stride 1 and padding 1. Each output channel has one bias. There is no pooling, grouping, dilation, or activation between the layers for this calculation.

Derive both output shapes, total trainable parameters, and multiply-accumulate counts, excluding bias additions. Determine the receptive-field width and input-space spacing of adjacent second-layer outputs. Explain why receptive-field extent at an image edge is not a count of real input pixels. State which numbers would change if a ReLU were inserted.

Get a starting hint

For each spatial dimension use floor((N + 2P - K) / S) + 1. Track receptive-field extent r and spacing j with r_new = r + (K - 1)j and j_new = Sj.

Compare your reasoning with the worked solution

The first spatial width is floor((32 + 4 - 5)/2) + 1 = 16, so its output is 16 by 16 by 8. Its filters contain 5 × 5 × 3 × 8 = 600 weights and eight biases, totaling 608 parameters. There are 2,048 output scalars, each using 75 multiply-accumulates: 153,600.

The second output is 16 by 16 by 16. Its parameter count is 3 × 3 × 8 × 16 + 16 = 1,168, and its multiply-accumulate count is 16 × 16 × 16 × 72 = 294,912. The combined totals are 1,776 parameters and 448,512 multiply-accumulates. These are arithmetic counts, not measured execution time; data movement and hardware scheduling matter.

Start with r = j = 1. The first layer gives r = 5 and j = 2. The second gives r = 5 + 2 × 2 = 9 and j = 2. A second-layer unit therefore has a nominal 9 by 9 input receptive-field extent. Near boundaries some locations represent padding, and paths through padded intermediate activations contribute no real image sample. An inserted ReLU leaves shapes, parameter counts, these convolution counts, and the maximum receptive-field extent unchanged. It adds comparisons and can make the input-dependent derivative vanish on some paths.

This is a teaching architecture, not a reconstruction of AlexNet's grouped layers or its reported performance.

Check your reasoning

• Obtains both shapes and 1,776 parameters without counting spatial reuse as new parameters.
• Derives 448,512 multiply-accumulates and a 9 by 9 receptive field with spacing 2.
• Distinguishes padding, potential dependence, and hardware runtime.

2. Finite model · allow 35 minutes

Test the dropout inference approximation

Take hidden activations h = (2, 4), output weights w = (1, -1/2), and no bias. Independently retain each hidden unit with probability q = 1/2. In this exercise use inverted dropout: retained activations are divided by q during training and left unchanged at inference.

Enumerate all four masks. Calculate the mean and variance of the preactivation z, then compare E[ReLU(z)] with ReLU(E[z]). Explain how this convention relates to the paper's training-time masking and test-time weight scaling. What does the example prove about treating deterministic inference as an exact average over nonlinear subnetworks?

Get a starting hint

The two unmasked contributions are 2 and -2. Under inverted dropout their retained contributions become 4 and -4. Every mask has probability 1/4.

Compare your reasoning with the worked solution

Masks (0,0), (1,0), (0,1), and (1,1) produce z = 0, 4, -4, and 0 respectively. Thus E[z] = 0 and E[z²] = (0 + 16 + 16 + 0)/4 = 8. The variance is 8 because the mean is zero. ReLU produces 0, 4, 0, and 0, so its expectation is 1. Deterministic inference gives ReLU(2 - 2) = 0. Therefore the expectation cannot generally be moved through a nonlinear activation.

The paper uses a different scaling convention: units are masked during training without this division, and outgoing weights are scaled for inference. At keep probability one half, that convention gives this example's training preactivations 0, 2, -2, 0 and deterministic preactivation 0. It likewise preserves the linear mean while failing to preserve the mean after ReLU. The conventions should not be mixed by scaling both training activations and inference weights.

The example is an exact four-case calculation, not evidence about AlexNet's measured classification error. It establishes that matching a linear expectation does not make a nonlinear network's deterministic output the exact ensemble average. For a full network, dropout changes learning as well as the distribution of activations; this calculation holds weights fixed and makes no claim that one convention gives the same optimization trajectory as another.

Check your reasoning

• Enumerates all four masks with correct probabilities and variance 8.
• Shows the nonlinear expectation mismatch 1 versus 0.
• Separates the exercise convention from the paper and avoids claiming an exact nonlinear ensemble.

Check your understanding

How many learned parameters are in the reading’s four-filter convolutional layer?

• 144
• 27,792
• 432
• 112

Explanation

Each filter has 27 weights and one bias; four filters give 112 parameters reused across positions.

A horizontal flip changes the correct class from left to right. What should a label-preserving augmentation pipeline do?

• Keep the old label because all flips preserve meaning.
• Reject that assumption or transform the label consistently.
• Apply the flip only to the test set.
• Count both predictions as automatically correct.

Explanation

An augmentation is valid only under the task’s actual label semantics.

Original paper and sources

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems, 2012.

Read the original paper →

Read the primary paper: NeurIPS

• ImageNet Classification with Deep Convolutional Neural Networks: NeurIPS