What is it?
Instead of people judging every answer, a short written "constitution" of principles is used: the AI critiques its own draft against them, rewrites it, and AI-made comparisons are used to train the model further. The rules become visible and editable, though they are only as good as the principles and the AI applying them.
Also known as: Constitutional AI, CAI, RLAIF, reinforcement learning from AI feedback, AI feedback, self-critique, model constitution, principle-based alignment.
Technical definition
Bai et al. (2022) describe a supervised phase (self-critique and revision against sampled principles, then fine-tuning on revisions) and an RL phase using AI-generated preference labels (RL from AI Feedback). Lee et al. (2023) compare RLAIF with RLHF on summarisation/dialogue tasks. Open issues: labeller bias, principle selection and conflicts, and evaluation of whether behaviour truly matches stated principles. Related practice: deliberative approaches and spec-style documents describing intended behaviour.
Entity type: technique · Freshness class: evolving.
When should I use it?
Useful where human labelling is costly or principles must be explicit.
Implementation patterns
- Write principles; sample critique prompts; revise; fine-tune.
- Generate AI preference pairs; train with DPO/RL.
- Human-audit samples of AI labels.
- Version the principle set.
Technologies and frameworks
Commonly associated: principle-driven critique/revision prompts, preference-pair generation with a labeller model. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Draft principles.
- Test critique/revision quality manually.
- Generate data at scale.
- Audit with humans.
- Evaluate behaviour on held-out scenarios.
Common failure modes
- Labeller model bias amplified in training.
- Vague or conflicting principles.
- Evasive/over-refusing behaviour.
- No human validation of AI labels.
Troubleshooting
- Model over-refuses — Add helpfulness principles and measure over-refusal.
- AI labels disagree with humans — Calibrate on a human-labelled subset; refine prompts.
Contrasted with
- RLHF (Reinforcement Learning from Human Feedback) — RLHF uses human preference labels; RLAIF uses model-generated labels guided by principles.
- AI Guardrails — Guardrails check outputs at runtime; constitutional methods change training.
Questions this page answers
- What is Constitutional AI?
- What is RLAIF?
- How does self-critique improve model safety?
- Can AI replace human feedback labelers?
Typical goals: reduce reliance on human labels; make alignment principles explicit.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Bai et al., Constitutional AI: Harmlessness from AI Feedback (2022) — arXiv:2212.08073
- Lee et al., RLAIF vs. RLHF (2023) — arXiv:2309.00267