EXPLAINER

GSM8K and Math Benchmarks

benchmarkfreshness: volatilesources not yet verified

GSM8K (grade-school maths word problems) and MATH (competition problems) are common reasoning benchmarks; GSM8K has been widely reported as close to saturated for strong models (a time-sensitive judgement), so harder maths sets are used to separate them.

Short answer: GSM8K tests multi-step arithmetic word problems; MATH tests competition-level problems. Strong models saturate GSM8K, and results depend on prompting (chain-of-thought), sampling and answer extraction — read protocols carefully.

What is it?

These are maths exams for AI. GSM8K has school-level word problems; MATH has tougher competition questions. Scores show how well a model reasons step by step, and are often used to showcase reasoning methods.

Also known as: GSM8K, MATH benchmark, math word problems benchmark, AIME benchmark, MATH-500, maths reasoning benchmark, grade school math.

Technical definition

GSM8K (Cobbe et al., 2021) has ~8.5k grade-school problems with natural-language solutions; MATH (Hendrycks et al., 2021) has competition problems with final answers. Evaluation requires answer extraction and normalisation; reported accuracy varies with few-shot Chain-of-Thought Prompting, Self-Consistency (Sampling and Voting) and Best-of-N Sampling settings. Harder sets and competition-derived evaluations are used as GSM8K saturates; contamination is a recognised risk (Benchmark Contamination and Data Leakage).

Entity type: benchmark · Freshness class: volatile.

When should I use it?

Reference points for reasoning research; supplement with your own domain tasks.

Implementation patterns

  • Fix prompt, shot count and extraction rules.
  • Report single-sample and aggregated (majority) accuracy separately.
  • Use harder, fresher sets for frontier comparison.

Technologies and frameworks

Commonly associated: lm-evaluation-harness, answer-normalisation scripts. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Choose dataset and split.
  2. Specify prompting protocol.
  3. Normalise answers.
  4. Report settings alongside results.

Common failure modes

  • Answer-format mismatches scored as wrong.
  • Mixing single-sample and majority-vote numbers.
  • Contamination.

Troubleshooting

  • Scores lower than reported — Check answer extraction and normalisation, sampling settings and prompt.

Contrasted with

Questions this page answers

  • What is GSM8K?
  • What is the MATH benchmark?
  • Are math benchmarks saturated?
  • How should I evaluate math reasoning?

Typical goals: interpret reasoning benchmark claims; evaluate maths reasoning of a model.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Saturation status and current best-known scores.

Canonical sources (unverified leads):

  • Cobbe et al., Training Verifiers to Solve Math Word Problems (GSM8K) (2021) — arXiv:2110.14168
  • Hendrycks et al., Measuring Mathematical Problem Solving With the MATH Dataset (2021) — arXiv:2103.03874