Attention Is All You Need

Content interaction replaces a sequential recurrence

The Transformer paper proposes an encoder-decoder architecture built around attention and position-wise transformations rather than sequence-aligned recurrence. Queries compare with keys to determine how values are mixed. Position information and decoder masking provide different constraints: one represents order, while the other prevents access to forbidden future tokens. The original architecture is a translation model with both encoder and decoder; it should not be silently replaced by the later decoder-only pattern in a historical explanation.

Original reconstruction: one query and two values

Take a query q=(√2,0), keys k1=(ln 3,0) and k2=(0,0), and scalar values v1=4 and v2=12. The key dimension is d_k=2. Scaled dot products are q·k1/√2=ln 3 and q·k2/√2=0. Applying softmax gives weights exp(ln 3)/(3+1)=3/4 and exp(0)/(3+1)=1/4.

The attention result is (3/4)×4+(1/4)×12=6. A high compatibility score selects how strongly a value contributes; the key itself is not the returned value. If the query becomes zero, both scores become zero, both weights become one half, and the output becomes eight.

Now assume the second value belongs to a future position forbidden by a causal mask. Excluding that score before normalization gives weights one and zero, so the output is four. Multiplying a future token’s value by zero after computing an unmasked softmax is not generally equivalent: its score would still have changed the normalization denominator and reduced the allowed value’s weight.

For numerical stability, softmax can subtract the largest allowed score before exponentiation. This leaves ratios unchanged because the same factor cancels from numerator and denominator. A row with no allowed positions requires an explicit implementation policy; applying ordinary normalization to an empty or fully excluded set is not a meaningful probability distribution.

Position and permutation are separate issues

For a fixed query, swapping complete key/value pairs leaves the weighted result unchanged when no position-dependent information or mask changes. That is useful for set-like retrieval but insufficient to distinguish every word order. Position encodings alter the representations so the model can make use of sequence position. A causal mask alone supplies a different structural restriction and is not interchangeable with the paper’s positional encoding.

Multiple heads apply distinct learned projections before attention and combine their outputs afterward. This supplies multiple learned interaction spaces, but a head need not correspond neatly to one human grammatical category. Large attention weights are part of the computation; they are not automatically a faithful explanation of why the final prediction changed.

Interpret the computational advantage precisely

During training with known target sequences and a correct mask, many position calculations can proceed in parallel. Autoregressive generation still ordinarily chooses a token before using that chosen token to predict the next. Removing recurrence from the layer does not make an unknown generated sequence available all at once.

For ordinary full attention over T positions, the pairwise score array contains T² entries per head. Doubling T quadruples this count, even though projection and other layer costs have different scaling. Actual implementations may avoid materializing the complete matrix or use other attention patterns, but those are additional techniques rather than a reason to erase the underlying interaction model.

A strong exercise verifies dimensions, masks, normalization, and permutation behavior on tiny arrays before scaling. It then measures quality and resource use under the task, because an elegant attention calculation alone does not establish useful language behavior.

import math
s = [math.log(2), 0.0, 0.0]
v = [1.0, 5.0, 9.0]
def output(logits):
    weights = [math.exp(x - max(logits)) for x in logits]
    return sum(a*b for a,b in zip(weights,v)) / sum(weights)
eps = 1e-6
for j in range(3):
    hi, lo = s.copy(), s.copy()
    hi[j] += eps
    lo[j] -= eps
    print((output(hi) - output(lo)) / (2*eps))

• Attention Is All You Need: NeurIPS