What is it?
Imagine marking a student's maths working line by line instead of only checking the final number. A process reward model does this for AI reasoning: it rates each step, so you can spot the first wrong step, steer a search toward good steps, or train a model with more detailed feedback.
The catch: step-by-step labels are costly to get, and the scoring model can itself be wrong.
Also known as: PRM, process supervision, step-level reward model, process-based reward, stepwise verifier, step verifier, Let's Verify Step by Step.
Technical definition
A PRM is a learned scorer r(x, s_1..s_t) over partial solutions, trained from step-level labels (human annotations such as PRM800K from Lightman et al., 2023, or automatically derived labels such as Monte Carlo rollouts in Math-Shepherd). Uses: rerank candidates by aggregating step scores, guide tree search (Search over Reasoning (Tree of Thoughts, MCTS, Beam Search)), and provide dense rewards in RL (Reinforcement Learning for Reasoning (RLVR and GRPO)). Risks include annotation cost, label noise from automatic labelling, domain shift, and reward over-optimisation (Reward Hacking and Over-Optimization).
Entity type: model · Freshness class: evolving.
When should I use it?
Consider a PRM when intermediate steps are verifiable, final-answer feedback is too sparse, or you need step-level error localisation. Use an ORM or programmatic check when only the end result matters.
Implementation patterns
- Rerank: score each candidate chain by min or product of step scores and choose the best.
- Guide search: expand steps with high PRM scores.
- Locate errors: show the first low-scoring step for debugging.
- Dense reward in RL where a verifiable end result is sparse.
Technologies and frameworks
Commonly associated: PRM800K dataset, Math-Shepherd, Monte Carlo rollout labelling. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define step segmentation.
- Collect or generate step labels and audit a sample by hand.
- Train the PRM and validate on held-out chains.
- Compare PRM reranking against ORM and majority vote.
- Monitor for over-optimisation as N or RL steps grow.
Common failure modes
- Step labels noisy or biased, especially when auto-generated.
- The PRM generalises poorly outside its training domain.
- Over-optimising against the PRM produces steps that look right to it but are wrong.
- Annotation cost makes it impractical for new domains.
- Ambiguity about what counts as a "step".
Troubleshooting
- PRM ranks plausible wrong chains highly — Add an outcome check (tests, ground truth) and retrain with hard negatives.
- Poor transfer to a new domain — Collect a small in-domain step-labelled set and fine-tune.
- Inconsistent step boundaries — Standardise step segmentation before labelling and scoring.
Contrasted with
- Outcome Reward Model (ORM) — ORM scores only the final result; PRM scores each step.
- PRM vs ORM: Process vs Outcome Reward Models — Side-by-side comparison with decision guidance.
Questions this page answers
- What is a process reward model?
- What is process supervision?
- How is a PRM trained?
- Why use a PRM instead of an outcome reward model?
Typical goals: understand step-level feedback for reasoning; choose between PRM and ORM; train or use a verifier.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Lightman et al., Let's Verify Step by Step (2023) — arXiv:2305.20050
- Uesato et al., Solving math word problems with process- and outcome-based feedback (2022) — arXiv:2211.14275
- Wang et al., Math-Shepherd (2023) — arXiv:2312.08935