Language models · 2020 · Tom B. Brown et al.
Language Models are Few-Shot Learners
Show that a sufficiently large autoregressive model can perform many tasks from instructions and examples in context without updating its weights.
How to read the original
- Read Approach and Evaluation, distinguishing zero-, one-, and few-shot settings.
- Read Measuring and Preventing Memorization Of Benchmarks before interpreting held-out scores.
- Read Limitations and compare task-specific results with broad claims about learning.
The prompt changes while weights remain fixed
The GPT-3 paper evaluates an autoregressive language model on tasks described through instructions and examples in context, without task-specific gradient updates during those evaluations. Its claim is empirical and varies across tasks and model scales. Few-shot prompting should therefore be distinguished from supervised fine-tuning, and a strong score on one benchmark should not be promoted into a guarantee that every new rule will be inferred reliably.
Original reconstruction: an unfamiliar label mapping
Create a toy classification task with invented labels. Define a hidden rule: even integers map to dax and odd integers to wug. A prompt contains examples 2→dax, 7→wug, and 4→dax, then asks for 9. A correct continuation is wug. The model’s parameters are unchanged, but its activations and next-token distribution depend on the supplied examples.
Now swap the label meanings consistently: even→wug and odd→dax. Test a new prompt with corresponding examples. A system that always emits familiar words associated with parity may fail this counterfactual mapping, while one using the demonstrated relation can adapt its output. This experiment does not prove an internal symbolic algorithm; it supplies behavioral evidence about sensitivity to context.
Include an ambiguous demonstration set containing only even inputs. Multiple rules fit those examples: parity, always-dax, or a threshold rule over the observed numbers. A wrong answer on a new odd number can reflect underdetermination of the prompt rather than an inability to represent parity. Write down competing hypotheses before interpreting the result.
Calculate a paired comparison
Suppose zero-shot prompting succeeds on 60 of 100 test items and five-shot prompting succeeds on 70. The difference is ten percentage points, not a ten-percent relative gain; relative to 60 successes, it is about 16.7%. To interpret paired behavior, suppose 55 items are correct under both, 15 only under five-shot, 5 only under zero-shot, and 25 under neither. The totals match 60 and 70, but the disagreement table reveals both improvements and regressions.
A useful uncertainty analysis resamples or tests paired items according to the experimental design rather than pretending that the two sets of results are unrelated. This small table is an original illustration, not a score from the paper. It also hides variation across task families, so inspect whether the extra successes cluster around examples similar to the demonstrations.
Control the prompt as part of the evaluated system
Choose demonstrations using only allowed development data. Record their ordering, formatting, label spelling, separator tokens, and the total context length. Randomize or vary these choices deliberately when studying robustness; do not silently choose the best test-set prompt after many attempts and report it as a single unbiased evaluation.
A model can exploit a familiar benchmark example encountered during pretraining. The absence of an exact string match does not rule out a near duplicate or leaked solution. Conversely, overlap does not establish that every observed success was memorized. Keep evidence and uncertainty separate, as the paper’s contamination discussion requires.
Bound the conclusion
An exact parser can grade the toy label task automatically, including extra text and malformed outputs according to a declared policy. Compare zero-shot, one-shot, and several demonstration counts at fixed model and decoding settings. Report cost, latency, and the effect of longer context alongside accuracy when relevant.
The important conceptual distinction is between changing parameters and conditioning a fixed computation on new data. Both can change behavior, but they have different persistence, resource use, and failure modes. A successful in-context example is a reason to investigate a capability under controlled variation, not a shortcut around the need for specifications and independent evaluation.
Reconstruction exercise
A paired test has 40 items correct under both prompts, 18 correct only under few-shot, 8 correct only under zero-shot, and 34 wrong under both. Compute both accuracies, the percentage-point difference, and the relative gain over zero-shot accuracy. Then design one control that distinguishes following the demonstrated label mapping from using a fixed familiar label convention.
Worked solution
There are 100 items. Zero-shot accuracy is (40+8)/100=48%, while few-shot accuracy is (40+18)/100=58%. The absolute difference is ten percentage points. Relative gain is (58−48)/48, approximately 20.83%. The disagreement cells matter because ten net additional successes conceal eighteen improvements and eight regressions. For the mapping control, use arbitrary labels and swap their meanings consistently in a second prompt while holding the underlying classification rule and held-out inputs fixed. A system that follows the demonstration should change its outputs accordingly. Include examples of both classes so the rule is not underdetermined, and keep test answers out of the demonstration selection process. Record prompt order and decoding settings. Even a successful label-swap control establishes behavior on this task, not a complete explanation of the model’s internal reasoning or a guarantee of general instruction following.
Test the limits and transfer the idea
1. Statistical reasoning · allow 45 minutes
Audit a prompting improvement on paired items
Two fixed prompting methods are tested on the same 100 independent items. Both answer 40 correctly and both miss 50. The new prompt alone succeeds on eight items; the old prompt alone succeeds on two. Calculate both accuracies and their difference.
Under a null model in which each of the ten discordant items is equally likely to favor either prompt, compute the exact two-sided sign-test probability of a split at least as uneven as 8 versus 2. Use math.comb, not a statistical package. Explain why the result is not the probability that the new method is useless, and identify what changes if these were the best results selected from 50 prompt variants.
Get a starting hint
Condition on ten discordant items. Under the stated null, the number favoring the new prompt is Binomial(10,1/2); add the two tails k ≤ 2 and k ≥ 8.
Compare your reasoning with the worked solution
The old prompt answers 40 + 2 = 42 items correctly; the new answers 40 + 8 = 48. Accuracy rises from 42% to 48%, an absolute difference of six percentage points. The paired comparison depends on the ten changed outcomes, not two unrelated samples of size 100.
The exact two-sided probability is 2 × [C(10,0)+C(10,1)+C(10,2)]/2^10 = 2(1+10+45)/1024 = 112/1024 = 0.109375.
from math import comb
p_value = 2 * sum(comb(10,k) for k in range(3)) / 2**10
assert p_value == 0.109375
At a preselected 0.05 threshold this test would not reject its null. That does not prove the prompts have equal quality, nor does 0.109375 assign a posterior probability to the claim that improvement is absent. It is a tail probability for the stated randomized-outcome model, conditional on discordance. The observed six-point gain and its uncertainty should both be reported.
If the prompt was selected as the best of 50 variants on these same items, the simple single-comparison interpretation ignores selection. Reserve a separate final test set or account for the search procedure. Correlated questions, subjective grading, and benchmark contamination can also invalidate or weaken the interpretation.
The data and test here are original teaching material, not GPT-3's published measurements. They develop the paper's evaluation distinction between prompting conditions without claiming that a small prompt comparison establishes general reasoning ability.
Check your reasoning
- Calculates 42%, 48%, and a six-percentage-point paired gain.
- Derives and computes the exact two-sided probability 0.109375.
- Explains null-model, selection, and generalization limits without misreading a p-value.
2. Finite hypothesis analysis · allow 40 minutes
Find what demonstrations leave undecided
A hypothetical in-context task maps integers 0 through 9 to labels D or W. Consider only these four candidate rules: E labels even integers D; T labels integers greater than 5 D; A always labels D; and L labels integers less than 5 D. Every other case receives W.
Filter the candidate set after demonstrations (2,D), (4,D), and then (7,W). List all remaining rules and choose a new input that distinguishes them. Explain why repeated demonstrations that do not distinguish the remaining rules cannot identify the intended mapping. Propose a held-out test that separates following an arbitrary label mapping from relying only on familiar label meanings.
Get a starting hint
Treat each demonstration as a constraint, not an instruction to change model weights. Tabulate every surviving rule's prediction on each unused input.
Compare your reasoning with the worked solution
After (2,D), T is eliminated because 2 is not greater than 5. E, A, and L survive. Demonstration (4,D) eliminates none of those three, since all predict D at 4. At (7,W), A is eliminated, leaving E and L: both predict W at 7.
Input 6 distinguishes them because E predicts D while L predicts W. Inputs 1 and 3 also distinguish them, with the opposite disagreement. Inputs 0 and 8 have different usefulness: at 0 both remaining rules predict D, while at 8 E predicts D and L predicts W. A complete tabulation exposes which extra evidence can actually reduce uncertainty.
For a mapping-following evaluation, choose arbitrary label strings, provide demonstrations sufficient to specify the intended rule within a declared task family, and test unused inputs. Repeat with the label meanings swapped, using matched inputs and comparable demonstrations. Score both conditions and record errors rather than assuming one successful completion reveals the internal mechanism. Avoid including the held-out answers in the prompt.
This is a finite identifiability exercise, not a model of GPT-3's actual hypothesis space. A language model may draw on many priors and representations outside these four rules. In-context conditioning supplies tokens while model parameters remain fixed during inference in the paper's setup; that is different from gradient-based task fine-tuning. A correct answer on one ambiguous example cannot by itself reveal whether the model inferred the intended function, exploited a prior, or guessed.
Check your reasoning
- Filters the candidates to E and L and gives a distinguishing unused input.
- Explains why redundant demonstrations fail to resolve ambiguity.
- Designs a held-out relabeling test and limits claims about model internals.
Check your understanding
What changes during the paper’s ordinary few-shot evaluation?
- The pretrained weights through task-specific gradient descent.
- The output weights through fitting a new classifier to the examples.
- The context supplied to the fixed model.
- The embedding weights through a supervised demonstration update.
Explanation
Task examples condition inference without the evaluation-time parameter updates used in fine-tuning.
Accuracy rises from 48% to 58%. What is the absolute increase?
- 10 percentage points.
- 20.83 percentage points.
- 10 correct answers per token.
- 58% relative gain.
Explanation
Percentage-point change subtracts the two percentages; relative gain divides by the baseline.
Original paper and sources
Tom B. Brown and coauthors. Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 2020.
Read the original paper →Read the primary paper: arXiv