What is it?
An outcome reward model is a grader for finished answers. It looks at the end result — "was the final answer right?" — and gives a score. It ignores whether the working was sound.
That is cheap and simple to get labels for (you only need to know the right answer), but it can reward a lucky guess or flawed reasoning that happened to land on the correct answer.
Also known as: ORM, outcome supervision, outcome-based reward, answer verifier, final-answer verifier, outcome-based feedback.
Technical definition
An ORM is trained to predict final-answer correctness (or preference) from a complete solution; Cobbe et al. (2021) trained such verifiers to rerank GSM8K samples. In RL it supplies a sparse terminal reward. Compared with process supervision it needs only outcome labels, but credit assignment across steps is weak and reasoning that reaches correct answers by faulty means is rewarded (Uesato et al., 2022 compare outcome- and process-based feedback). In domains with exact checkers, a programmatic verifier replaces the learned ORM (Reinforcement Learning for Reasoning (RLVR and GRPO)).
Entity type: model · Freshness class: evolving.
When should I use it?
Use when the final result is checkable and the path matters less, or labelling effort must stay low.
Implementation patterns
- Exact check: compare to a known answer or run unit tests, no learning required.
- Learned ORM reranking of N samples.
- Terminal reward in RL for verifiable tasks.
- Mixed: ORM for ranking, PRM for diagnosing errors.
Technologies and frameworks
Commonly associated: programmatic verifiers (unit tests, exact match), learned verifiers, GSM8K / MATH answer checkers. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Pick a programmatic checker when possible.
- Otherwise label final-answer correctness for a diverse set of solutions.
- Train and validate the ORM, checking calibration.
- Use it for reranking before using it in RL.
- Audit selected samples for flawed reasoning.
Common failure modes
- Rewards correct answers reached through invalid reasoning.
- Sparse signal makes RL slow or unstable on long chains.
- Learned ORM exploited by plausible-but-wrong answers.
- Answer-format mismatches scored as wrong.
Troubleshooting
- Right answers with bad reasoning are rewarded — Add step checks, or a PRM, or audit samples by hand.
- Format mismatches scored wrong — Normalise answers (units, formatting) in the checker.
- RL learns slowly — Use curriculum, easier problems first, or denser rewards.
Contrasted with
- Process Reward Model (PRM) — PRM scores steps; ORM scores only the end result.
- PRM vs ORM: Process vs Outcome Reward Models — Comparison with decision guidance.
Questions this page answers
- What is an outcome reward model?
- How is an ORM different from a PRM?
- Can I use unit tests instead of a learned ORM?
- What are the downsides of outcome-only rewards?
Typical goals: understand outcome-only feedback; choose a verifier for reranking; compare outcome and process supervision.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Cobbe et al., Training Verifiers to Solve Math Word Problems (2021) — arXiv:2110.14168
- Uesato et al., Solving math word problems with process- and outcome-based feedback (2022) — arXiv:2211.14275