EXPLAINER

MMLU (Massive Multitask Language Understanding)

benchmarkfreshness: volatilesources not yet verified

MMLU is a multiple-choice benchmark spanning 57 subjects used to compare the breadth of knowledge in language models. It is widely reported but now largely saturated and contains known label issues.

Short answer: MMLU tests multiple-choice knowledge across 57 subjects. High scores signal broad knowledge but the benchmark is near saturation for top models, has noted errors, and is sensitive to prompt format; harder variants (such as MMLU-Pro) exist.

What is it?

MMLU is like a giant multiple-choice university exam covering many subjects, from history to maths. Models are often compared by their MMLU score. Today the best models score so high that it no longer separates them well.

Also known as: MMLU, massive multitask language understanding, MMLU score, MMLU-Pro, GPQA, multiple choice benchmark, what does MMLU measure.

Technical definition

Hendrycks et al. (2020) introduced MMLU with questions from 57 subject areas; evaluation is typically few-shot accuracy. Known issues include annotation errors in some subsets, answer-order and prompt-format sensitivity, and contamination risk (Benchmark Contamination and Data Leakage). MMLU-Pro (Wang et al., 2024) increases difficulty and answer options; other related sets target specific domains (GPQA, Rein et al., 2023).

Entity type: benchmark · Freshness class: volatile.

When should I use it?

Reference point for knowledge breadth; not a selection criterion alone.

Implementation patterns

  • Use as a coarse knowledge check only.
  • Compare scores only under identical prompts and shot counts.
  • Prefer harder, newer variants for frontier models.

Technologies and frameworks

Commonly associated: lm-evaluation-harness, HELM. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Check which variant and protocol a score uses.
  2. Reproduce with a standard harness.
  3. Complement with task-specific tests.

Common failure modes

  • Treating a few points' difference as meaningful.
  • Ignoring format sensitivity.
  • Equating multiple-choice knowledge with real-world helpfulness.

Troubleshooting

  • Cannot reproduce a reported score — Match the harness version, prompt template and shot count.

Contrasted with

Questions this page answers

  • What is MMLU?
  • What does an MMLU score mean?
  • Is MMLU still a good benchmark?
  • What is MMLU-Pro?
  • What is GPQA?

Typical goals: interpret an MMLU score; find harder alternatives.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Saturation status, current top scores and variant adoption.

Canonical sources (unverified leads):

  • Hendrycks et al., Measuring Massive Multitask Language Understanding (2020) — arXiv:2009.03300
  • Wang et al., MMLU-Pro (2024) — arXiv:2406.01574
  • Rein et al., GPQA (2023) — arXiv:2311.12022