FREE LESSON · LLMs & agents · 1 OF 7
How Transformers Work: Tokens and Attention
A transformer predicts one token at a time: Tokens, embeddings, attention, and residual computation
Text becomes vectors whose interactions are learned.
A tokenizer maps text fragments to discrete IDs. Embeddings map IDs and positions to vectors. In a decoder-only language model, causal self-attention lets each position mix information from itself and permitted earlier positions; feed-forward layers transform each position; residual connections carry and refine representations through many blocks. A final projection produces next-token probabilities. Other transformer architectures, such as bidirectional encoders, use different attention masks and objectives.
The model manipulates learned statistical representations, not a database of sentences or a symbolic world model by default.
Attention is content-addressed communication
Each position produces a query, key, and value. In causal attention, it can use keys and values at its own and earlier positions. Query–key similarity determines normalized weights, and the weighted values deliver context. Multiple heads can learn different interaction patterns. Attention does not by itself guarantee human-readable reasoning or faithful attribution.
A useful mechanism can support a behaviour without uniquely explaining that behaviour.
Sources and further reading
- Attention Is All You Need: Vaswani et al. · primary paper
- Retrieval-Augmented Generation: Lewis et al. · primary paper
- AI Risk Management Framework: NIST