What is it?
Both are "graders" for reasoning. A PRM grades every step; an ORM grades only the end result. Step grading tells you where it went wrong and gives richer feedback, but needs expensive step-by-step labels. Result grading is cheap but blunt.
Also known as: PRM vs ORM, process vs outcome reward model, process supervision vs outcome supervision, step reward vs final reward, which reward model for reasoning.
Technical definition
| PRM | ORM | |
|---|---|---|
| Scores | Each step / prefix | Complete solution |
| Labels | Step-level (human or automatic) | Final-answer correctness |
| Signal | Dense | Sparse |
| Error location | Yes | No |
| Label cost | High | Low |
| Typical risk | Noisy or biased step labels; over-optimisation | Rewards flawed reasoning with right answers |
| Fit | Search, error diagnosis, dense RL rewards | Reranking, terminal RL reward, verifiable tasks |
Published comparisons on maths tasks differ in what they find: Uesato et al. (2022, GSM8K) report that outcome-based supervision reached similar final-answer error rates with less labelling, while process-based supervision (or reward models that emulate it) was needed for correct reasoning steps; Lightman et al. (2023, MATH) report that process supervision significantly outperformed outcome supervision. Results depend on domain, model scale and labelling method and should not be assumed to transfer.
Entity type: concept · Freshness class: evolving.
When should I use it?
Pick programmatic outcome checks first, an ORM when labels must be cheap, and a PRM when step-level signal justifies the labelling cost.
Implementation patterns
- Start with a programmatic outcome check; add a PRM only where step diagnosis matters.
- Use ORM to rerank and PRM to explain failures.
- Validate any learned scorer on held-out data before using it in RL.
Technologies and frameworks
Commonly associated: PRM800K, programmatic verifiers, Math-Shepherd. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Can the final answer be checked exactly? Use that first.
- Do you need error localisation or search guidance? Consider a PRM.
- Estimate labelling cost for each.
- Pilot on 100–500 examples.
- Audit for over-optimisation as you scale.
Common failure modes
- Choosing a PRM when no one can afford step labels.
- Using an ORM where right-for-wrong-reasons answers are unacceptable.
- Assuming published results transfer to your domain.
Troubleshooting
- Unsure which to pick — Run both on a small labelled set and compare reranking accuracy and cost.
Contrasted with
- Process Reward Model (PRM) — Step-level scorer.
- Outcome Reward Model (ORM) — Final-answer scorer.
Questions this page answers
- What is the difference between PRM and ORM?
- Should I use a process or outcome reward model?
- Is process supervision better than outcome supervision?
- Which reward model is cheaper to train?
Typical goals: choose a verifier approach; understand the trade-off between label cost and signal quality.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Lightman et al., Let's Verify Step by Step (2023) — arXiv:2305.20050
- Uesato et al., Solving math word problems with process- and outcome-based feedback (2022) — arXiv:2211.14275