FREE LESSON · LLMs & agents · 1 OF 4
How Transformers Work: Tokens and Attention
A transformer predicts one token at a time — Tokens, embeddings, attention, and residual computation
Text becomes vectors whose interactions are learned.
A tokenizer maps text fragments to discrete IDs. Embeddings map IDs and positions to vectors. Self-attention lets each position form a weighted mixture of information from permitted earlier positions; feed-forward layers transform each position; residual connections carry and refine representations through many blocks. A final projection produces next-token probabilities.
The model manipulates learned statistical representations, not a database of sentences or a symbolic world model by default.
Attention is content-addressed communication
Each position produces a query; earlier positions produce keys and values. Query–key similarity determines normalized weights, and the weighted values deliver context. Multiple heads can learn different interaction patterns. Attention does not by itself guarantee human-readable reasoning or faithful attribution.
A useful mechanism can support a behaviour without uniquely explaining that behaviour.