What is it?
After basic training, people compare pairs of the model's answers and say which is better. A "reward model" learns to predict those preferences, and the AI is then trained to produce answers the reward model scores highly — while being kept from drifting too far from its original behaviour.
Also known as: RLHF, reinforcement learning from human feedback, reward model training, human feedback training, InstructGPT method, reward model, PPO RLHF, how ChatGPT was trained.
Technical definition
Pipeline popularised by Christiano et al. (2017) and applied to language models in summarisation work (Stiennon et al., 2020) and InstructGPT (Ouyang et al., 2022). Components: preference data collection, reward model training (Bradley–Terry loss), RL optimisation (PPO) with a KL penalty against a reference policy to limit over-optimisation. Known limitations catalogued by Casper et al. (2023): evaluator error and bias, reward misspecification, reward hacking (Reward Hacking and Over-Optimization), mode collapse and sycophancy (Sycophancy in Language Models). Alternatives and complements: Preference Optimization (Aligning Models from Human or AI Preferences) methods such as Direct Preference Optimization (DPO), and AI feedback (Constitutional AI and RLAIF (AI Feedback)). RLHF uses RL as a component but is typically classified as an alignment technique.
Entity type: technique · Freshness class: evolving.
When should I use it?
Mostly the concern of model developers; consumers should understand its limitations.
Implementation patterns
- Train the reward model on diverse, well-guided comparisons.
- Keep a KL penalty; monitor reward vs held-out human eval.
- Refresh the reward model as the policy improves.
- Red-team the policy for reward-model exploits.
Technologies and frameworks
Commonly associated: TRL, OpenRLHF, DeepSpeed-Chat, reward-model datasets. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Write annotation guidelines.
- Collect comparisons.
- Train and validate the reward model.
- Run PPO with KL control.
- Evaluate with humans and held-out sets.
Common failure modes
- Reward hacking.
- Annotator disagreement and bias.
- Over-optimisation reducing quality.
- Heavy compute and engineering.
- Sycophancy and verbosity.
Troubleshooting
- Reward rises but human ratings fall — Increase KL, update the reward model with fresh comparisons, early-stop on human eval.
- Model flatters users — Add disagreement-appropriate pairs and sycophancy evals.
Contrasted with
- Direct Preference Optimization (DPO) — DPO skips the reward model and RL loop.
- Reinforcement Learning for Reasoning (RLVR and GRPO) — RLVR uses programmatic verifiable rewards, not human preference models.
Questions this page answers
- What is RLHF?
- How does RLHF work?
- What is a reward model?
- Why does RLHF cause sycophancy?
- What is the KL penalty in RLHF?
Typical goals: understand how assistants are trained to follow preferences; identify the failure modes of RLHF.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Christiano et al., Deep Reinforcement Learning from Human Preferences (2017) — arXiv:1706.03741
- Stiennon et al., Learning to summarize from human feedback (2020) — arXiv:2009.01325
- Ouyang et al., Training language models to follow instructions with human feedback (2022) — arXiv:2203.02155
- Casper et al., Open Problems and Fundamental Limitations of RLHF (2023) — arXiv:2307.15217