Language-model evaluation and evidence

Prerequisites: Conditional probability; sampling; classification metrics.

Learning objectives

• Compute negative log-likelihood and perplexity.
• Recognize contamination and selection effects.
• Design reproducible task evaluations.

An evaluation needs a claim

Specify what capability is being measured, which inputs represent the intended use, what counts as success, and how outputs are scored. A model can improve on next-token likelihood without improving a particular factual or reasoning task. It can also exploit artifacts in a benchmark that do not transfer to new inputs.

For observed tokens with conditional probabilities p₁ through pₙ, average negative log-likelihood is -(1/n) times the sum of log pᵢ. Perplexity is its exponential when natural logarithms are used. For two tokens with probabilities 1/2 and 1/8, their joint probability is 1/16; the average negative log-likelihood is log 4 and perplexity is 4. This is a likelihood measure over the specified tokenization, not a percentage of correct answers.

Preserve the evaluation boundary

Training on test examples, near duplicates, or their solutions contaminates evaluation. So can repeated model selection against a public test set. Separate development data from final assessment and document what is known about overlap. An unknown training corpus creates uncertainty; it does not justify assuming either perfect cleanliness or definite contamination.

For programming tasks, use held-out tests that include boundaries and adversarial cases. Running only the examples visible in a prompt rewards imitation. For mathematical tasks, distinguish correct final answers from valid derivations. For open-ended answers, a rubric should identify factual accuracy, completeness, and unsupported claims separately.

Count attempts and uncertainty

A score depends on the inference procedure. One sample, ten samples with an oracle choosing the best, and ten samples selected by a practical verifier are different systems. Record sampling parameters, prompts, tools, retries, stopping rules, and selection criteria. Report latency and cost alongside quality when they matter to the application.

If two systems differ by a few successes on a small test set, sampling variability may dominate the apparent gain. Pair evaluations on the same examples where possible, inspect disagreement cases, and use uncertainty estimates appropriate to the sampling design. A single aggregate can hide large differences across languages, difficulty levels, or user groups.

Model-based judges can help scale review, but their agreement with human criteria must be checked. They may favor verbosity or share blind spots with the model being judged. Preserve a human-audited sample and make the rubric explicit.

A useful evaluation is a reproducible argument: here are the inputs, outputs, scoring rules, uncertainty, and limits of the claim. A leaderboard position alone cannot establish that a system is reliable for an unfamiliar task.

Practice problem

Two tokens have probabilities 1/4 and 1/4. Compute perplexity. Then design a small evaluation for a code-explanation assistant that checks more than fluent writing. Include a failure case and a rule for comparing versions.

Worked solution

The joint probability is 1/16, the mean negative log-likelihood is log 4, and perplexity is 4. For code explanations, sample programs with known behavior, include edge cases such as aliasing and off-by-one loops, and ask for predictions plus reasoning. Score behavioral accuracy, identification of assumptions, and unsupported claims separately. Keep a held-out set, run both versions on identical prompts and settings, and manually review disagreements. A fluent explanation that predicts the wrong result fails behavioral accuracy. Report the sample size and uncertainty rather than treating a tiny score difference as conclusive.

Check your understanding

Perplexity 4 means what?

• Exactly 25% of answers are correct.
• The exponential of average negative log-likelihood is 4.
• Every generated answer has four words.
• The model has four parameters.

Explanation

Perplexity is a likelihood-derived measure, not task accuracy.

Why must retry counts be reported?

• Retries never affect results.
• They only change formatting.
• Multiple attempts and selection change the evaluated system and its resource use.
• They prove the benchmark is contaminated.

Explanation

A best-of-many procedure is different from a single-attempt system.

Primary reading

• Language Modeling from Scratch: Stanford CS336
• Attention Is All You Need: Vaswani et al.

Open the study and save your work →