What is it?
A powerful AI that does exactly what you literally said, or what looked rewarding during training, isn't necessarily doing what you meant. Alignment is the research and engineering work to close that gap — so models are helpful, don't deceive and avoid harmful behaviour.
Also known as: alignment, AI safety, value alignment, aligned AI, outer alignment, inner alignment, scalable oversight, helpful honest harmless.
Technical definition
Problem framings include outer alignment (is the specified objective what we want?), inner alignment (does the learned system pursue that objective?), reward misspecification and gaming (Reward Hacking and Over-Optimization), scalable oversight (supervising systems that exceed evaluator ability), robustness and honesty. Methods: Instruction Tuning, RLHF (Reinforcement Learning from Human Feedback), Preference Optimization (Aligning Models from Human or AI Preferences) (including Direct Preference Optimization (DPO)), Constitutional AI and RLAIF (AI Feedback), adversarial testing (Red-Teaming AI Systems), and Mechanistic Interpretability. Surveys such as Amodei et al. (2016) and the RLHF limitations paper (Casper et al., 2023) document open problems. Claims about long-term risk are contested and outside this entry's scope.
Entity type: concept · Freshness class: evolving.
When should I use it?
Relevant to anyone training, fine-tuning or deploying models, at least as a risk checklist.
Implementation patterns
- Layer defences: aligned model + guardrails + permissions + monitoring.
- Evaluate refusals, over-refusals and honesty on dedicated test sets.
- Red-team before release and continuously.
- Document limits in model cards.
Technologies and frameworks
Commonly associated: RLHF pipelines, DPO and variants, Constitutional AI, red-teaming programmes, interpretability tooling. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define intended behaviour in writing.
- Build evaluation sets for helpfulness, honesty, harm and over-refusal.
- Choose training approach.
- Red-team.
- Monitor in production and iterate.
Common failure modes
- Treating alignment as a one-time training step.
- Over-refusal making the model unhelpful.
- Sycophancy mistaken for helpfulness.
- Reward models exploited during optimisation.
- Evaluating only on known attack patterns.
Troubleshooting
- Model agrees with false user claims — Add sycophancy tests; train on disagreement-appropriate data; evaluate honesty.
- Too many refusals — Measure over-refusal rates and rebalance preference data.
Contrasted with
- AI Guardrails — Guardrails are runtime checks around a model; alignment shapes the model's own behaviour.
- AI Governance — Governance is organisational and legal oversight; alignment is a technical property and research field.
Questions this page answers
- What is AI alignment?
- How are chatbots aligned?
- What is the difference between AI safety and alignment?
- What is reward hacking?
- Is alignment solved?
Typical goals: understand what alignment means in practice; map alignment techniques.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Amodei et al., Concrete Problems in AI Safety (2016) — arXiv:1606.06565
- Casper et al., Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023) — arXiv:2307.15217
- Bai et al., Training a Helpful and Harmless Assistant with RLHF (2022) — arXiv:2204.05862