Language models · 2020 · Tom B. Brown et al.
Language Models are Few-Shot Learners
Show that a sufficiently large autoregressive model can perform many tasks from instructions and examples in context without updating its weights.
The central move
Show that a sufficiently large autoregressive model can perform many tasks from instructions and examples in context without updating its weights.
Why it had to exist
NLP systems were commonly fine-tuned separately for each benchmark. Scaling raised the possibility that one pretrained model could infer task behavior from the prompt itself.
Where it leads
Autoregressive scale → in-context learning → prompting and general-purpose language-model interfaces.