# John Schulman > 1986– · Computer scientist; OpenAI co-founder; Chief Scientist at Thinking Machines Lab > > **Recorded contribution:** TRPO and PPO; reinforcement learning from human feedback; OpenAI co-founder; Thinking Machines Lab Chief Scientist ## How to use this dossier Read for a causal chain, not a hero story: inherited problem → contribution → mechanism → downstream capability → limit. Then close the page and complete the reconstruction exercise from memory. ## 1. Historical orientation American computer scientist John Schulman co-founded OpenAI and led influential reinforcement-learning research including trust region policy optimization (TRPO) and proximal policy optimization (PPO). He also co-authored work on learning from human preferences that became part of the RLHF lineage. Schulman left OpenAI for Anthropic in 2024 and later joined Thinking Machines Lab, whose site listed him as chief scientist at verification time; affiliations remain historical snapshots. The shared principle behind TRPO and PPO is that a policy update should be useful without moving so far that data collected under the old policy becomes misleading. PPO's clipped surrogate converted that trust-region intuition into a simpler minibatch objective practical for large neural policies. ## 2. The problem inherited Policy-gradient updates can improve an estimated objective while changing behavior so drastically that performance collapses, and desired language-model behavior is difficult to specify as a differentiable reward. ## 3. The central contribution Schulman led TRPO and PPO, practical methods that constrain policy updates, and helped connect preference feedback to learned reward optimization. ## 4. Reconstruct the mechanism 1. Collect trajectories from the current policy and estimate advantages for its actions. 2. Compute a probability ratio between the candidate and old policy. 3. Clip or constrain that ratio so one update cannot move behavior too far. 4. For preference learning, fit a reward model from human comparisons and optimize the policy against it while monitoring divergence. ## 5. What changed downstream - PPO became a widely used reinforcement-learning algorithm across games, robotics, and language-model post-training. - Preference-based training helped make RLHF a central technique for shaping assistant behavior. - The methods provided a bridge from policy-gradient theory to robust engineering practice and later became components in preference-based post-training pipelines for language models. ## 6. Attribution, limits, and uncertainty - TRPO, PPO, and preference-learning papers have multiple authors and build on policy gradients, trust regions, and reward modeling. - Clipping is a heuristic, and optimizing a learned preference reward can exploit annotator bias or model error rather than achieve true alignment. - Stable optimization of a proxy reward is not alignment: the preference dataset, reward model, sampling policy, and evaluator population jointly determine which behavior is reinforced. ## 7. Reconstruction lab Run a two-action bandit with an unconstrained policy update and a clipped PPO update. Then replace the true reward with noisy pairwise preferences and show reward-model overoptimization. Vary the clipping threshold and preference-noise rate independently, plotting true reward and learned reward so stability cannot be mistaken for objective validity. ## 8. Evidence trail - [Trust Region Policy Optimization](https://proceedings.mlr.press/v37/schulman15.html) — ICML / PMLR - [Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347) — arXiv - [Deep Reinforcement Learning from Human Preferences](https://arxiv.org/abs/1706.03741) — NeurIPS - [Thinking Machines Lab founding team](https://thinkingmachines.ai/) — Thinking Machines Lab --- *Research checked 2026-08-09. Dates, roles, and claims about living people are historical snapshots. Linked sources remain the authority; this dossier is original instructional synthesis.*