What is it?
Ask the model the same hard question several times, with some randomness, and see which final answer comes up most often. If most attempts agree, that answer is more likely right than a single attempt.
It works best when answers are short and comparable (a number, a label) and the model's mistakes are random rather than consistent.
Also known as: majority voting, majority vote sampling, self consistency decoding, maj@n, maj@k, voting over samples, ensemble sampling.
Technical definition
Self-consistency (Wang et al., 2022) replaces greedy decoding with sampling of N reasoning paths at temperature > 0, extracts each final answer, and returns the plurality answer. It requires an answer-extraction step and answers that can be normalised for comparison. Cost scales about linearly with N. It cannot correct systematic errors the model makes in most samples, and it is less applicable to free-form outputs where "agreement" is undefined.
Entity type: technique · Freshness class: evolving.
When should I use it?
Use for tasks with discrete answers (maths, classification, multiple choice, extraction) where you can afford several calls. Avoid for open-ended writing.
Implementation patterns
- Fixed N: draw 5–20 samples at moderate temperature, normalise answers, take the plurality.
- Early stopping: stop drawing once one answer has an unassailable lead.
- Agreement as confidence: use the vote share to flag low-agreement answers for human review.
- Structured answers: extract the answer via a schema so votes compare cleanly.
Technologies and frameworks
Commonly associated: temperature / top-p sampling, regex or structured-output answer extraction, vLLM n-sampling. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Make the final answer extractable (schema or labelled line).
- Pick temperature that gives diverse but sensible samples.
- Start with N=5; test N=10, 20 on your cases.
- Normalise before voting.
- Compare accuracy and cost per correct answer against single-pass (How to Evaluate AI Systems and Outputs).
Common failure modes
- Different surface forms of the same answer ("12", "twelve", "12.0") counted as different votes.
- All samples share the same systematic mistake, so the majority is confidently wrong.
- Temperature too low, producing near-identical samples and no real diversity.
- Cost multiplied N-fold without a measured accuracy gain.
Troubleshooting
- Votes split across many answers — Normalise answers, or accept the task is too hard for this model and add a verifier.
- No accuracy gain — Check agreement vs correctness on your test set; if errors are consistent, change method.
- Too slow — Parallelise samples and use early stopping.
Contrasted with
- Best-of-N Sampling — Best-of-N needs a scorer to pick one sample; self-consistency uses agreement between samples and no scorer.
- Chain-of-Thought Prompting — CoT is a single reasoning chain; self-consistency aggregates many.
Questions this page answers
- What is self-consistency in LLMs?
- How does majority voting improve LLM accuracy?
- How many samples should I use for self-consistency?
- Does self-consistency work for free-form text?
Typical goals: improve accuracy of a single-answer task; estimate how reliable an answer is; decide how many samples to draw.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022) — arXiv:2203.11171