Tokenization and the unit of prediction

Prerequisites: Strings, dictionaries, and conditional probability.

Learning objectives

• Explain subword tokenization.
• Distinguish bytes, characters, and tokens.
• Compare likelihood only with compatible units.

Text is encoded before prediction

A language model usually predicts token identifiers from a finite vocabulary. Tokens may represent whole words, word fragments, punctuation, or byte sequences. A token is not generally a word, and a character is not generally one byte. Unicode characters can require multiple bytes in UTF-8; a visible grapheme can also contain multiple Unicode code points.

A tokenizer is part of the model contract. The same visible text can produce different token sequences under different vocabularies or normalization rules. A model's embedding and output matrices refer to particular token IDs. Swapping tokenizers without changing the model does not preserve the meaning of those IDs.

Trace a simplified merge procedure

Byte-pair encoding begins with a base symbol vocabulary and learns merges of adjacent symbol pairs. In a toy corpus, suppose the pair ('l','o') is most frequent. Replacing that pair with 'lo' changes the counts of neighboring pairs. A later merge might combine ('lo','w') into 'low'. The learned merge order matters when encoding a new string.

For a simplified input represented as [l,o,w,e,r], applying those two merges yields [low,e,r]. If a subsequent learned merge combines ('e','r'), it becomes [low,er]. These tokens are artifacts of frequency and the merge procedure. They do not establish that the model has acquired grammatical concepts of roots and suffixes.

Real tokenizers add pre-tokenization, normalization, special tokens, and explicit treatment of bytes or unknown symbols. Study those rules before assuming exact round-trip recovery. Lossy normalization can intentionally make multiple original strings map to the same representation.

Probabilities depend on the representation

The probability of a token sequence factors into conditional probabilities: P(t₁,...,tₙ)=product of P(tᵢ given earlier tokens). Negative log-likelihood sums the negative logarithms of those terms. A per-token average depends on how the tokenizer partitions the text. Comparing per-token perplexity across different tokenizers can therefore be misleading.

Tokenization also affects context budgets, handling of unusual identifiers, multilingual efficiency, and exact string tasks. A model may need several prediction steps to emit a number that a person sees as one unit. Reliable evaluation should use the task's actual success criterion, such as exact decoded output or a parsed structured result, rather than assuming a token-level improvement settles it.

Keep tokenization, model inference, and text decoding as separate stages in your implementation. Inspect their intermediate representations when debugging. Many apparent reasoning failures begin with an incorrect assumption about what the model actually received.

Practice problem

Using base symbols [a,b,a,b,c], apply a learned merge (a,b)→ab and then (ab,c)→abc. Show the sequence after each merge. Explain why two models with different tokenizers cannot be fairly ranked by per-token perplexity alone.

Worked solution

The first merge replaces the two non-overlapping occurrences of (a,b), giving [ab,ab,c]. The second merge applies to the final pair, giving [ab,abc]. Per-token perplexity averages surprise over a tokenizer-dependent number of units. A tokenizer that uses longer tokens changes both the sequence length and individual token probabilities. Compare compatible likelihood units, such as normalized byte-level description length when appropriate, and evaluate the actual downstream task on the same examples. The encoding and normalization contracts must also be documented.

Check your understanding

Which statement is generally correct?

• Every token is one Unicode character.
• A token can represent several characters or part of a byte sequence.
• Every word always has one token.
• Token IDs have universal meanings across models.

Explanation

The vocabulary determines the mapping; tokens and characters are different units.

Why does merge order matter?

• Earlier merges change the adjacent symbols available to later merges.
• All merges happen on the original string independently.
• Merge order changes Unicode itself.
• It removes the need for a vocabulary.

Explanation

A merge replaces symbols and therefore changes subsequent candidate pairs.

Primary reading

• Language Modeling from Scratch: Stanford CS336
• Attention Is All You Need: Vaswani et al.

Open the study and save your work →