ARTIFICIAL INTELLIGENCE
InstructGPT and learning from human feedback
The 2022 InstructGPT paper connected demonstrations, preference modeling, and reinforcement learning while showing why preferred answers still need factual checks.
Research checked
Newest cited publication
A general agent–environment diagram introduces actions and reward feedback. InstructGPT adds the human comparisons, reward modeling, and policy-training procedure explained in the article.
Megajuice · Image source · CC0 1.0
A language model trained to continue text is not automatically trained to produce the answer a person wants. An instruction can be followed by an answer, another instruction, a discussion of the task, or a plausible but irrelevant continuation. InstructGPT's 2022 research made that gap experimentally concrete. The approach combined demonstrations of desired responses with comparisons between alternatives, then used a learned reward to adjust a pretrained model. This historical landmark concerns a training pipeline and an evaluation, rather than proof that a system acquired human values. Its significance becomes clearer when the three learning stages are separated and the feedback process is treated as a measurement instrument with its own uncertainty, coverage, and potential failure modes.
A smaller preferred model was a conditional result
Training language models to follow instructions with human feedback was first submitted on 4 March 2022. Long Ouyang and colleagues fine-tuned GPT-3 using labeler demonstrations, trained a reward model from output rankings, and applied reinforcement learning. In human evaluations on the study's prompt distribution, a 1.3-billion-parameter InstructGPT model was preferred to the 175-billion-parameter GPT-3 baseline. The result established that this feedback pipeline could improve evaluated instruction-following behavior; it did not establish that the smaller model had greater capability on every task.
An analogy is a technically skilled employee receiving examples of how a report should be delivered. Knowing relevant facts and presenting the requested report are different requirements. A model can likewise have useful predictive structure while producing an inconvenient response format. The analogy concerns observable behavior only. It does not imply that the model has an employee's understanding, intentions, or responsibilities. The experimental question is whether a training procedure changes outputs in a way that people prefer under specified instructions.
The pipeline asks people two different questions
Demonstrations provide complete examples of desired responses. Comparisons ask which candidate response is preferable. The resulting reward model estimates preferences, and the policy is optimized against that estimate. The InstructGPT paper also includes a penalty for departing from the supervised starting policy and a variant mixing in pretraining updates. Its authors reported remaining mistakes and limitations in whose preferences the data represented. These qualifications belong beside the result because the learned target depends on the annotation process.
Consider a request to explain a simple calculation in two sentences. A demonstrator can write an example that satisfies both the arithmetic and the format. A comparison worker might prefer a more polished answer that accidentally uses three sentences. Neither observation alone fully specifies the task. Several comparisons can also disagree because workers prioritize brevity, completeness, or tone differently. Treating those judgments as noisy measurements encourages clear instructions and held-out checks, rather than treating every ranking as an exact reading of an invisible universal preference.
The central feedback idea preceded InstructGPT
Deep reinforcement learning from human preferences, first submitted on 12 June 2017, learned reward signals from comparisons between short behavior segments. Paul Christiano and colleagues tested the approach in Atari and simulated robotics settings, including behavior trained without exposing the environment's reward function to the learner. That work supplied an important earlier demonstration of preference-based reward learning. The task setting differs from language generation, so its experimental outcomes should not be transferred directly to the later instruction-following models.
The common structure is a separation between observing a preference and making future decisions. In a fictional sorting simulation, a person might choose which of two short sequences better protects fragile objects. A learned scorer can then guide many additional actions without asking the person about every action. That leverage is useful precisely because it extrapolates beyond the labeled comparisons. It is also the source of risk: the scorer may reward a superficial cue that correlated with careful handling in the labeled examples but fails in a new situation.
Summarization made the proxy problem visible
Learning to summarize from human feedback was submitted on 2 September 2020. Nisan Stiennon and colleagues trained a preference model from summary comparisons and used it to optimize a summarization policy. Their experiments included Reddit TL;DR summaries and transfer to news summarization. The paper also examined what happened as reward optimization increased, including the mismatch between the learned reward and human judgments under stronger optimization. This is directly relevant historical evidence that increasing a preference model's score is not identical to improving the output people actually judge.
Suppose a source text has three facts: an inspection was scheduled, it was delayed, and a replacement date is unknown. A polished summary that invents a replacement date may seem complete while changing the content. A preference worker needs access to the source to catch that error. An evaluator who sees only the summary is measuring something different. Good experimental design therefore specifies the information available to annotators. Readability, factual support, and usefulness can overlap, but none should silently serve as a substitute for all the others.
PPO regulates updates rather than validating truth
InstructGPT used Proximal Policy Optimization, or PPO. The PPO paper, submitted on 20 July 2017, introduced policy-gradient objectives designed to support multiple optimization passes over sampled experience while discouraging overly large policy changes. Its clipped objective limits the incentive associated with certain probability-ratio changes; clipping is not an absolute bound on all parameter changes. The original evaluations concerned control and game benchmarks. PPO is an optimization method, and it cannot turn an inaccurate reward into a factual or ethically complete objective.
A deliberately simplified calculation shows the role of the ratio. Suppose an action had probability 0.10 under the sampling policy and probability 0.13 under a proposed update. The ratio is 1.3. With a positive advantage estimate and a clipping threshold of 0.2, the clipped surrogate uses a ceiling of 1.2 for the beneficial ratio contribution. This toy calculation omits the rest of the training system. It explains why repeatedly increasing an already favored action need not keep improving that surrogate, without claiming that PPO guarantees safe behavior.
A reward difference is not a truth probability
Preference models are often trained so that a higher score makes one candidate more likely to win a comparison. For a numerical illustration, a logistic comparison model assigns a candidate with a score advantage of one a predicted win probability of about 0.731. That is a prediction about the comparison labels under the model, not a 73.1 percent probability that the candidate's claims are true. The distinction remains even if the score is precise and the optimizer reliably increases it.
A useful evaluation would keep separate records for instruction compliance, factual support, response style, and disagreement among raters. It would examine unfamiliar prompts and cases where an attractive answer conflicts with source material. Those proposed checks follow from the measurement problem, rather than asserting a new result about the 2022 models. If the reward model and the final evaluator share the same blind spot, agreement between them can look convincing while providing little independent evidence. Evaluation needs a source of information beyond the signal being optimized.
The achievement was a practical change in behavior
The historical significance of InstructGPT lies in joining existing ideas into a broad instruction-following experiment with a clear human-preference result. Demonstrations establish a response pattern, comparisons estimate desirable differences, and reinforcement learning changes the response distribution. Each stage has a distinct role and a distinct way to fail. Calling the whole process alignment can be useful shorthand, provided it does not erase the narrower experimental definition or the people and criteria behind the feedback.
A careful replication would preserve the prompt distribution, annotation instructions, candidate-generation settings, reward-model training, policy-update budget, and final evaluation procedure. It would report uncertainty and failure cases alongside average preference. The lasting lesson is methodological: improving a model's behavior requires a target closer to the intended use than raw text continuation, but every substitute target deserves scrutiny. Human feedback can make a system more useful without supplying a complete factual verifier or resolving every disagreement about what useful behavior should be.
Sources & further reading
Primary papers, project documentation, and research releases used in this article. Publication dates below refer to the sources themselves.
- Training language models to follow instructions with human feedbackarXivPublished Checked
- Deep reinforcement learning from human preferencesarXivPublished Checked
- Learning to summarize from human feedbackarXivPublished Checked
- Proximal Policy Optimization AlgorithmsarXivPublished Checked
FOLLOW THE QUESTION
Study engineering foundations