# Tri Dao > ~1993– · Computer Scientist, Creator of FlashAttention > > **Recorded contribution:** FlashAttention — memory-efficient attention; crucial to modern LLM training efficiency ## How to use this dossier Read for a causal chain, not a hero story: inherited problem → contribution → mechanism → downstream capability → limit. Then close the page and complete the reconstruction exercise from memory. ## 1. Historical orientation Vietnamese-American computer scientist Tri Dao works at the boundary of machine learning and systems. During his Stanford doctorate he led FlashAttention, an exact attention algorithm organized around accelerator memory hierarchy rather than fewer arithmetic operations. He later co-developed the Mamba selective state-space architecture, became a Princeton professor, and co-founded Together AI. FlashAttention derives speed from the memory hierarchy. Instead of materializing the full attention-score matrix in high-bandwidth memory, it tiles queries, keys, and values into faster on-chip memory, performs online softmax bookkeeping, and recomputes selected quantities during the backward pass. ## 2. The problem inherited Transformer attention spent substantial time moving large intermediate matrices between high-bandwidth memory and fast on-chip memory, so nominal FLOP counts did not explain real speed or memory use. ## 3. The central contribution Dao led FlashAttention's IO-aware tiled exact attention and co-developed efficient long-sequence state-space methods. ## 4. Reconstruct the mechanism 1. Partition query, key, and value matrices into tiles that fit fast on-chip SRAM. 2. Stream key/value tiles, computing score blocks without materializing the full attention matrix in high-bandwidth memory. 3. Maintain online softmax statistics so normalization remains exact across tiles. 4. Recompute selected intermediates during backpropagation to trade inexpensive arithmetic for reduced memory traffic. ## 5. What changed downstream - FlashAttention substantially improved Transformer training and inference efficiency on modern GPUs. - The work made hardware-aware algorithm design a central part of modern ML research. - The algorithm made longer contexts and larger batches practical and established input/output complexity—not arithmetic count alone—as a central design criterion for machine-learning kernels. ## 6. Attribution, limits, and uncertainty - FlashAttention has five original authors and depends on accelerator kernels, online softmax, and attention predecessors. - Speedups depend on shapes and hardware; exact attention retains quadratic arithmetic, and faster models can increase total compute and environmental use. - Exact mathematical equivalence does not guarantee identical floating-point results, performance depends on hardware and shapes, and faster attention can encourage larger models whose total resource use still rises. ## 7. Reconstruction lab Implement naive attention and a tiled online-softmax version for small matrices. Count reads and writes between simulated slow and fast memory and verify outputs match. Count bytes moved as well as operations for naive and tiled attention, then find a small shape where kernel overhead makes the theoretically superior method slower. ## 8. Evidence trail - [Tri Dao](https://www.cs.princeton.edu/people/profile/td8762) — Princeton University - [FlashAttention](https://arxiv.org/abs/2205.14135) — NeurIPS / arXiv - [Tri Dao publications](https://tridao.me/) — Tri Dao --- *Research checked 2026-08-09. Dates, roles, and claims about living people are historical snapshots. Linked sources remain the authority; this dossier is original instructional synthesis.*