What is it?
Both start from "which answer is better?" data. RLHF turns that into a scoring model and then trains the AI by trial and reward. DPO skips the scoring model and trains directly on the comparisons. DPO is easier to run; RLHF has more flexibility and more ways to go wrong.
Also known as: DPO vs PPO, RLHF vs DPO, DPO or RLHF, should I use DPO or RLHF, DPO versus PPO, direct preference optimization vs reinforcement learning from human feedback, is DPO better than RLHF.
Technical definition
| RLHF (PPO-style) | DPO | |
|---|---|---|
| Stages | SFT → reward model → RL | SFT → DPO |
| Models in memory | Policy, reference, reward, (value) | Policy, reference |
| Data use | Online sampling scored by reward model | Offline preference pairs (on-policy variants possible) |
| Stability | Sensitive to RL hyperparameters | Simpler, sensitive to β and data |
| Reward hacking exposure | Via learned reward model | Implicit; data and length biases |
| Flexibility | Any scalar reward, including programmatic | Needs pairwise (or compatible) data |
| Classification | RL on a learned reward | Preference optimisation (supervised-style loss) |
Empirical comparisons are mixed and depend on data and tuning; do not assume one dominates.
Entity type: concept · Freshness class: evolving.
When should I use it?
DPO by default for pairwise data on modest budgets; RLHF/online RL for reward-driven or on-policy needs.
Implementation patterns
- Start with DPO for a first aligned model; move to online/RL methods if you need on-policy reward optimisation or programmatic rewards.
- Evaluate both with the same held-out win-rate set.
Technologies and frameworks
Commonly associated: TRL, OpenRLHF, Axolotl. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Check your data format (pairs vs scalar rewards).
- Estimate hardware.
- Run DPO baseline.
- Evaluate.
- Escalate if needed.
Common failure modes
- Choosing RLHF complexity without need.
- Assuming DPO removes data-quality problems.
Troubleshooting
- Unsure which to choose — Run a small DPO baseline first; justify RL only if it plateaus.
Contrasted with
- Direct Preference Optimization (DPO) — Direct method.
- RLHF (Reinforcement Learning from Human Feedback) — Reward-model + RL method.
Questions this page answers
- What is the difference between DPO and RLHF?
- Is DPO better than PPO?
- Do I need a reward model for DPO?
- Which is cheaper, DPO or RLHF?
Typical goals: pick an alignment method; understand trade-offs between online and offline preference methods.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Rafailov et al., DPO (2023) — arXiv:2305.18290
- Ouyang et al., InstructGPT (2022) — arXiv:2203.02155
- Ivison et al., Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback (2024) — arXiv:2406.09279