# Richard Sutton > 1956– · Computer Scientist, AI Researcher > > **Recorded contribution:** Reinforcement learning; TD learning; Reinforcement Learning textbook; "The Bitter Lesson" ## How to use this dossier Read for a causal chain, not a hero story: inherited problem → contribution → mechanism → downstream capability → limit. Then close the page and complete the reconstruction exercise from memory. ## 1. Historical orientation Richard Sutton developed temporal-difference learning, helped formalize reinforcement learning with Andrew Barto, advanced policy-gradient and prediction methods, and argued in “The Bitter Lesson” that general methods leveraging computation tend to outlast systems dominated by hand-coded domain knowledge. This work belongs to the history of making machine behavior depend on representations, evidence, objectives, and evaluation rather than on a separate hand-written rule for every case. The chronology is used causally: it connects the inherited constraint to an implementable mechanism and then to later reuse, instead of treating fame, job title, or eventual market success as the explanation. ## 2. The problem inherited An agent often receives delayed consequences rather than labeled correct actions, so it must assign credit through time while learning from incomplete experience. The first-principles difficulty is not simply “make a machine intelligent”: it is to specify what is represented, where evidence comes from, how a procedure changes with evidence, and what observation would count as failure. ## 3. The central contribution Temporal-difference learning updates a value estimate toward reward plus the next state’s current estimate, combining sampling with bootstrapping. Its importance therefore lies in an inspectable learning or search mechanism, not in an anthropomorphic claim about the system understanding as a person does. ## 4. Reconstruct the mechanism 1. Define states, actions, rewards, discounting, and the policy that generates experience. State the task, representation, and success measure before selecting an algorithm. 2. Observe a transition and compute the temporal-difference error between current estimate and one-step target. Trace where evidence or feedback changes internal state; do not hide learning behind a product label. 3. Update the visited value or action value by a controlled step size. Run the resulting procedure on a small case where every intermediate value can be inspected. 4. Change exploration, reward, function approximation, or environment dynamics and test instability, bias, and reward hacking. Change the data, objective, or environment and locate the first place behavior ceases to generalize. ## 5. What changed downstream - TD learning became foundational in reinforcement learning, game-playing systems, control, and modern actor–critic methods; the Sutton–Barto text unified the field’s teaching language. - Downstream systems inherited both a reusable method and a warning: benchmark performance depends on the data-generating process and evaluation contract. - The transferable first-principles lesson is to separate the artifact named in “Reinforcement learning; TD learning; Reinforcement Learning textbook; "The Bitter Lesson"” from the mechanism, surrounding institution, and evidence that allowed later systems to depend on it. ## 6. Attribution, limits, and uncertainty - Sutton’s work is deeply collaborative, especially with Barto. “The Bitter Lesson” is a historical argument, not a theorem; compute-scaling methods still embody human choices about objectives, data, architectures, and evaluation. - Later success does not retroactively prove that every historical motivation, cognitive analogy, or priority claim was correct. - The subject is living or the registry has no death year; current titles and institutional affiliations are treated as dated snapshots verified on 2026-08-09, not permanent identity claims. ## 7. Reconstruction lab Implement TD(0) for a five-state random walk and compare learned values with Monte Carlo estimates. Alter the reward location and step size, then explain the new error dynamics. Report the representation, objective, update/search rule, held-out test, and one deliberately adversarial example. ## 8. Evidence trail - [Reinforcement Learning: An Introduction](http://incompleteideas.net/book/the-book-2nd.html) — Richard Sutton and Andrew Barto - [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) — Richard Sutton - [Richard Sutton](https://en.wikipedia.org/wiki/Richard_Sutton) — Wikipedia contributors · overview and bibliography - [Richard Sutton structured identity record](https://www.wikidata.org/wiki/Q7329307) — Wikidata contributors · CC0 --- *Research checked 2026-08-09. Dates, roles, and claims about living people are historical snapshots. Linked sources remain the authority; this dossier is original instructional synthesis.*