# Rewon Child

> ~1990– · AI Researcher
>
> **Recorded contribution:** Sparse Transformer; scaling up GPT architectures at OpenAI

## How to use this dossier

Read for a causal chain, not a hero story: inherited problem → contribution → mechanism → downstream capability → limit. Then close the page and complete the reconstruction exercise from memory.

## 1. Historical orientation

AI researcher Rewon Child was first author of the 2019 Sparse Transformer paper with Scott Gray, Alec Radford, and Ilya Sutskever and later worked on deep variational autoencoders. Sparse Transformer introduced fixed sparse attention factorizations and memory-saving techniques for much longer sequences. Public biographical sources are sparse, so the registry's approximate birth year is not treated as verified. Sparse Transformer reduced attention cost by giving each position a structured subset of earlier positions rather than every predecessor. Strided and fixed patterns allowed information to travel across local and long-range routes over multiple layers, while recomputation and custom kernels reduced memory pressure.

## 2. The problem inherited

Full self-attention uses memory and computation quadratic in sequence length, preventing direct modeling of long text, image, and audio sequences.

## 3. The central contribution

Child led Sparse Transformer, demonstrating structured attention patterns that reduced asymptotic cost and enabled substantially longer generative contexts.

## 4. Reconstruct the mechanism

1. Replace the full attention matrix with chosen local and strided connectivity patterns.
2. Ensure information can propagate between distant positions through compositions of sparse layers.
3. Recompute attention activations during backpropagation to save memory.
4. Train deep autoregressive models on images, audio, and text and compare likelihood and coherence.

## 5. What changed downstream

- The work helped establish efficient attention as a major research area.
- It demonstrated that long-context dependencies could be approximated without connecting every pair in every layer.
- The work helped establish efficient attention as a major research area and showed that useful long-context modeling depends on hardware-aware implementations as well as asymptotic sparsity.

## 6. Attribution, limits, and uncertainty

- Sparse Transformer has four authors and many efficient-attention predecessors and successors.
- Fixed sparsity can miss task-relevant interactions, kernel efficiency may differ from asymptotic theory, and the registry's birth estimate is unresolved.
- A fixed pattern encodes assumptions about which interactions matter, effective communication may require many layers, and nominal FLOP savings can disappear when sparse operations map poorly to accelerators.

## 7. Reconstruction lab

Draw full, local, and strided attention graphs for sixteen positions. Calculate edges and shortest dependency paths and construct one relation the sparse pattern delays. Construct a dependency that the sparse pattern cannot connect in one layer, calculate the required path length, and compare theoretical operations with measured runtime.

## 8. Evidence trail

- [Generating Long Sequences with Sparse Transformers](https://arxiv.org/abs/1904.10509) — arXiv
- [Very Deep VAEs Generalize Autoregressive Models](https://arxiv.org/abs/2011.10650) — arXiv

---

*Research checked 2026-08-09. Dates, roles, and claims about living people are historical snapshots. Linked sources remain the authority; this dossier is original instructional synthesis.*