Sequence modeling · 2017 · Ashish Vaswani et al.
Attention Is All You Need
Replace recurrence at the core of sequence transduction with parallel content-based interaction, while adding explicit position information.
The central move
Replace recurrence at the core of sequence transduction with parallel content-based interaction, while adding explicit position information.
Why it had to exist
Recurrent models forced a long sequential training path and made distant dependencies difficult. Existing attention still sat beside recurrent encoders and decoders.
Where it leads
Parallel self-attention → Transformers → BERT, GPT, multimodal models, and today’s language interfaces.