EXPLAINER

Preference Optimization (Aligning Models from Human or AI Preferences)

conceptfreshness: volatilesources not yet verified

Preference optimisation is the family of methods that tune a model using comparisons of better versus worse responses — RLHF, DPO and related algorithms. It is an alignment technique family, distinct from ordinary reinforcement learning.

Short answer: Given pairs (or rankings) of preferred vs rejected responses, these methods push the model toward preferred behaviour. RLHF learns a reward model and then uses RL; DPO and relatives optimise directly on the preference pairs with a supervised-style loss. Both live under alignment, not general RL.

What is it?

People find it much easier to say which of two answers is better than to write the perfect answer. Preference optimisation uses those "A is better than B" judgements to steer a model toward responses people prefer. Different algorithms do this in different ways.

Also known as: preference tuning, preference learning, alignment training, preference-based fine-tuning, human preference optimization, offline preference optimization, RLHF alternatives, preference data.

Technical definition

The standard setup assumes a Bradley–Terry-style model of preferences over responses y_w ≻ y_l to prompt x, with a KL constraint to a reference policy. Pipelines: (a) RLHF — fit a reward model to preferences, then optimise the policy with RL such as PPO (RLHF (Reinforcement Learning from Human Feedback), Proximal Policy Optimization (PPO)); (b) direct methods that optimise the policy on preference data without an explicit reward model or an online sampling RL loop — DPO (Rafailov et al., 2023; Direct Preference Optimization (DPO)), IPO (Azar et al., 2023), KTO (Ethayarajh et al., 2024; uses binary desirable/undesirable labels), ORPO (Hong et al., 2024) and SimPO (Meng et al., 2024); (c) AI-generated preferences (Constitutional AI and RLAIF (AI Feedback)). Classification note: methods in (b) are not reinforcement learning in the ordinary sense, though they derive from an RL objective; verifiable-reward RL (Reinforcement Learning for Reasoning (RLVR and GRPO)) is a separate family.

Entity type: concept · Freshness class: volatile.

When should I use it?

Applies when tuning open models for helpfulness/safety or style preference.

Implementation patterns

  • SFT first, then preference optimisation on pairs.
  • Collect or generate chosen/rejected pairs from the model's own samples (on-policy) when possible.
  • Hold out preference data for evaluation.
  • Mix in safety and over-refusal pairs.

Technologies and frameworks

Commonly associated: TRL (DPOTrainer, KTOTrainer, ORPOTrainer), OpenRLHF, Axolotl, LLaMA-Factory, preference datasets. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Prepare an SFT model.
  2. Collect preference pairs with clear guidelines.
  3. Pick a method (start with DPO for simplicity).
  4. Train with a reference model and conservative learning rate.
  5. Evaluate quality, safety and over-refusal.

Common failure modes

  • Noisy or biased preference labels.
  • Length and style bias in preferences.
  • Distribution mismatch between preference data and model outputs.
  • Over-optimisation reducing diversity or quality.
  • Skipping SFT and starting from a weak base.

Troubleshooting

  • Outputs become verbose — Control for length in data and loss; add length-normalised variants.
  • Quality regresses on other tasks — Lower beta/learning rate, add regularisation data, evaluate broadly.

Contrasted with

  • Reinforcement Learning — General RL optimises reward from environment interaction; preference optimisation learns from comparative judgements about responses.
  • Instruction Tuning — SFT imitates demonstrations; preference optimisation learns from comparisons.

Questions this page answers

  • What is preference optimization?
  • What are the alternatives to RLHF?
  • What is preference data?
  • What is KTO?
  • Is DPO reinforcement learning?

Typical goals: understand how chat models are tuned to human preferences; choose an alignment method.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Which variants are current best practice and their comparative results.
  • Library APIs and defaults.

Canonical sources (unverified leads):

  • Ouyang et al., Training language models to follow instructions with human feedback (2022) — arXiv:2203.02155
  • Rafailov et al., Direct Preference Optimization (2023) — arXiv:2305.18290
  • Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024) — arXiv:2402.01306
  • Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences (IPO) (2023) — arXiv:2310.12036
  • Hong et al., ORPO (2024) — arXiv:2403.07691
  • Meng et al., SimPO (2024) — arXiv:2405.14734
Learning path: Alignment: from RLHF to DPO (step 5 of 10)
← RLHF (Reinforcement Learning from Human Feedback)Direct Preference Optimization (DPO) →