Training Language Models to Follow Instructions with Human Feedback

How to read the original

1. Read Methods and experimental details, especially High-level methodology.
2. Read the preference-model and policy-training descriptions while distinguishing labels, rewards, and the reference policy.
3. Read Discussion and separate improvements on the evaluated prompt distribution from universal alignment claims.

Capability and preferred behavior are different objectives

The paper studies supervised demonstrations, preference modeling, and reinforcement learning as stages for changing a pretrained language model’s behavior. Human comparisons train a reward proxy, and policy optimization uses that proxy while penalizing divergence from a reference policy. The reported preference result concerns a particular prompt and evaluator distribution; it does not establish that smaller tuned models outperform larger base models on every capability or that learned preferences guarantee factual truth.

Original reconstruction: learn a comparison score

Suppose a prompt has candidate answers A and B, and the labeler prefers A. Let the reward model assign scalar scores r(A)=1.2 and r(B)=0.2. A common pairwise model assigns preference probability σ(r(A)−r(B))=σ(1), approximately 0.7311. The negative log-likelihood of the observed preference is about 0.3133.

If both scores increase by five, the difference remains one and the modeled preference probability is unchanged. Pairwise comparisons therefore do not by themselves determine an absolute reward origin. A score of 6.2 is not a calibrated probability that an answer is correct, nor is it a universal unit of helpfulness.

Now imagine the labeler prefers a longer, polished answer that contains a subtle factual error. The reward model can learn a pattern correlated with that preference. Optimizing the model’s score more strongly may amplify the pattern rather than repair the error. Label quality, representativeness, and evaluation independent of the training proxy are therefore part of the learning problem.

Work a simplified regularized policy objective

Use a toy policy with only two possible complete answers. The reference policy assigns probabilities (0.5,0.5); a candidate policy assigns (0.8,0.2). Suppose rewards are 2 for the first answer and 0 for the second. Expected reward rises from one under the reference to 1.6 under the candidate.

The KL divergence from candidate to reference is 0.8 ln(0.8/0.5)+0.2 ln(0.2/0.5), approximately 0.1927 nats. Under objective expected_reward−β KL, β=1 yields about 1.4073 for the candidate, above the reference’s one. With β=4, the candidate objective is about 0.8290, below the reference. The coefficient changes the trade-off between pursuing the proxy and moving away from the reference.

This calculation is a deliberately simplified two-action example, not the full PPO training algorithm or a reproduction of the paper’s measured results. A KL penalty is a soft objective term, not a hard guarantee that every output remains safe or close to an individually preferred response. Its effect depends on the coefficient, estimator, optimization, and distribution.

Distinguish the training stages in an audit

Demonstration learning supplies examples of desired responses. Preference learning supplies comparisons among responses. Policy optimization changes which responses the model generates. Each stage can fail differently: demonstrations can be narrow, comparisons can disagree, the reward model can generalize poorly, and optimization can exploit proxy errors.

Evaluate on prompts withheld from training and reward-model selection. Keep factual correctness, instruction adherence, style preference, and harmful behavior distinguishable in the rubric. An aggregate preference win can conceal a regression in one of those dimensions. Include disagreement analysis and test unfamiliar prompt categories instead of assuming the training distribution represents every user.

The transferable lesson is to make the optimization target and evidence explicit. Human feedback is data produced by people under instructions and constraints. Treat it as a valuable but fallible signal, then ask which behaviors were measured, whose judgments shaped the proxy, and what independent evidence would reveal a failure that the reward model itself would miss.

Reconstruction exercise

Two answer scores are 0.7 and 0.7, and the first answer is preferred. Compute the pairwise preference probability and negative log-likelihood. Then compare reference probabilities (0.5,0.5) with candidate probabilities (0.75,0.25), rewards (2,0), and β=1. Compute expected reward, KL divergence, and the regularized objective. Explain why a higher objective is not proof of factual reliability.

Worked solution

Equal scores have difference zero, so the modeled preference probability is σ(0)=0.5 and the negative log-likelihood is −ln(0.5), approximately 0.6931. The candidate’s expected reward is 0.75×2+0.25×0=1.5. Its KL divergence to the reference is 0.75 ln(1.5)+0.25 ln(0.5), approximately 0.1308 nats. With β=1, the regularized objective is approximately 1.3692, above the reference value one. This means the candidate is favored by the stated toy reward and regularization objective. It does not establish that the reward encodes truthfulness, that human comparisons were error-free, or that the candidate generalizes to new prompts. A separate factual assessment must examine claims against reliable evidence on held-out examples. The calculation also abstracts away policy-optimization details, sequence length, sampling variance, and token-level implementation choices, so it should not be presented as a full training recipe.

Test the limits and transfer the idea

1. Constrained optimization · allow 55 minutes

Solve a tiny reward-versus-reference objective

For one prompt, allow only two responses. The reference policy is p=(1/2,1/2), rewards are r=(ln 3,0), and a candidate policy q maximizes J(q)=Σqᵢrᵢ - βΣqᵢln(qᵢ/pᵢ), subject to qᵢ≥0 and Σqᵢ=1. Use natural logarithms.

Derive the optimal policy for general β>0, evaluate β=1 and β=2, and compute J at β=1. Show why adding the same constant to every reward leaves the optimizer unchanged. Explain which parts of the paper's training setup this finite objective omits.

Get a starting hint

Use a Lagrange multiplier for normalization. The stationary condition gives ln(qᵢ/pᵢ) = rᵢ/β plus a constant shared by all responses.

Compare your reasoning with the worked solution

Differentiating with the normalization multiplier gives rᵢ - β(ln(qᵢ/pᵢ)+1) + λ = 0. Therefore qᵢ is proportional to pᵢ exp(rᵢ/β). With Z = Σpᵢ exp(rᵢ/β), the normalized optimum is qᵢ*=pᵢ exp(rᵢ/β)/Z. Positive reference probabilities and finite rewards give an interior solution. More generally, J(q)=β ln Z - β KL(q||q*), which proves global optimality since KL is nonnegative.

For β=1, Z=(3+1)/2=2 and q*=(3/4,1/4). Its expected reward is (3/4)ln3, approximately 0.823959. Its KL from the reference is (3/4)ln(3/2)+(1/4)ln(1/2), approximately 0.130812. Thus J=ln2, approximately 0.693147. For β=2, q₁*=√3/(1+√3), approximately 0.633975, and q₂* is the remainder. A stronger penalty moves the optimum closer to the reference.

Adding constant c to both rewards multiplies all unnormalized weights by exp(c/β), which cancels in normalization. It adds c to J but does not change the maximizing policy.

This finite exact optimization is an original teaching model for the paper's reward and reference-policy tension. It omits token sequences, sampled policy-gradient updates, PPO approximations, learned reward errors, prompt distributions, and the additional pretraining objective in PPO-ptx. Its closed-form optimum does not imply that the full training process reaches an analogous exact solution or that high modeled reward establishes factual correctness.

Check your reasoning

• Derives the normalized exponential tilt and proves optimality.
• Obtains (3/4,1/4), the β=2 policy, and J=ln2.
• Explains reward-shift invariance and the omitted training mechanisms.

2. Proof and loss analysis · allow 45 minutes

Test what a scalar reward cannot represent

For one prompt, equal numbers of pairwise judgments prefer A over B, B over C, and C over A. A scalar reward model predicts P(X preferred to Y)=sigmoid(r_X-r_Y). Can all three predicted probabilities exceed 1/2?

For one judgment of each type, minimize the summed negative log-likelihood over the three real-valued rewards. Prove your minimum rather than only trying r_A=r_B=r_C numerically. Explain how heterogeneous annotators or context can produce such a cycle and why fitting a scalar reward does not transform the judgments into objective truth.

Get a starting hint

The three reward differences sum to zero. The function f(t)=ln(1+exp(-t)) is convex, so Jensen's inequality applies to their average.

Compare your reasoning with the worked solution

Each predicted probability exceeds 1/2 exactly when its reward difference is positive. The three differences are d₁=r_A-r_B, d₂=r_B-r_C, and d₃=r_C-r_A. Their sum is zero, so all three cannot be positive. A scalar ordering cannot represent a strict directed cycle.

The loss is f(d₁)+f(d₂)+f(d₃), with f(t)=ln(1+exp(-t)). Its second derivative is positive for finite t, so f is strictly convex. Jensen's inequality gives the loss at least 3f((d₁+d₂+d₃)/3)=3f(0)=3ln2, approximately 2.079442. Equality requires all differences to be equal; because their sum is zero, each is zero. Thus any equal reward triple minimizes the loss and predicts 1/2 for all three pairs. Adding the same constant to all rewards does not change anything.

The cycle can arise from differing annotator priorities, omitted context, or noisy judgments; it need not mean every individual holds the same cyclic preferences. If contexts are distinguishable inputs, a richer model may represent different conditional rankings. Pooling them into one prompt-level scalar can hide that structure.

This equal-weight three-response problem is an original analysis of the paper's pairwise reward-learning form, not a claim about a specific observed cycle in its data. The likelihood objective fits recorded comparisons under a model. It does not establish that a response is true, ethically correct, or unanimously preferred.

Check your reasoning

• Uses the zero-sum reward differences to rule out three probabilities above 1/2.
• Proves the global loss minimum 3ln2 and characterizes all equal-score minimizers.
• Separates model expressivity, heterogeneous judgments, and objective correctness.

Check your understanding

Both reward scores increase by the same constant. What happens to the pairwise preference probability?

• It doubles.
• It becomes exactly one.
• It becomes the constant.
• It stays unchanged because the score difference is unchanged.

Explanation

The sigmoid comparison depends on the difference, not the absolute offset.

What does increasing β in the toy objective do?

• Makes the reward model’s preference scores calibrated truth probabilities.
• Makes divergence from the reference more costly in the objective.
• Forces the learned policy to equal the reference after every update.
• Converts the regularized objective into a fixed per-answer divergence bound.

Explanation

The coefficient changes an optimization trade-off; it does not establish every desired behavior.

Original paper and sources

Long Ouyang and coauthors. Training Language Models to Follow Instructions with Human Feedback. Advances in Neural Information Processing Systems, 2022.

Read the original paper →

Read the primary paper: OpenAI

• Training Language Models to Follow Instructions with Human Feedback: OpenAI