EXPLAINER

HumanEval and Code Generation Benchmarks

benchmarkfreshness: evolvingsources not yet verified

HumanEval measures whether a model can write correct Python functions from docstrings, scored by running unit tests (pass@k). It is small and saturated; newer code benchmarks test harder and more realistic tasks.

Short answer: HumanEval = 164 hand-written Python problems checked by unit tests, reported as pass@k. It is saturated and narrow; use richer evaluations such as SWE-bench or LiveCodeBench for realistic coding ability, and your own test suite for your stack.

What is it?

HumanEval gives a model a function description and checks whether the code it writes passes hidden tests. It was a standard way to compare coding ability, but modern models solve most of it, so harder tests are now used.

Also known as: HumanEval, pass@k, pass at k, MBPP, EvalPlus, code generation benchmark, coding benchmark, LiveCodeBench.

Technical definition

Chen et al. (2021, the Codex paper) introduced HumanEval and the pass@k metric (probability that at least one of k samples passes). Extensions: HumanEval+ (EvalPlus; more tests), MBPP, multilingual variants, and LiveCodeBench (time-split problems, Benchmark Contamination and Data Leakage). Execution-based evaluation requires sandboxing generated code (Code Execution and Sandboxing for AI Agents).

Entity type: benchmark · Freshness class: evolving.

When should I use it?

Historical reference for function-level code generation.

Implementation patterns

  • Run generated code in a sandbox with timeouts.
  • Report pass@1 with fixed sampling settings.
  • Add your own repository tests.
  • Include edge-case and hidden tests, and review a sample of passing solutions for readability and security, not only correctness.

Technologies and frameworks

Commonly associated: EvalPlus, bigcode-evaluation-harness, sandboxed executors. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Choose benchmark(s) matching your language and task type, and note which are saturated or contaminated.
  2. Sandbox execution with timeouts and no credentials.
  3. Fix sampling parameters and report pass@1 separately from pass@k.
  4. Complement with your own repository tests, linting and code review criteria.

Common failure modes

  • Running untrusted generated code unsandboxed.
  • Over-trusting a saturated benchmark.
  • Comparing pass@k computed with different n and k.

Troubleshooting

  • Scores differ between papers — Check pass@k estimator settings, temperature and prompt format.

Contrasted with

Questions this page answers

  • What is HumanEval?
  • What does pass@k mean?
  • Is HumanEval still useful?
  • How do I evaluate a coding assistant?

Typical goals: interpret coding benchmark scores; choose evaluations for a coding assistant.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Chen et al., Evaluating Large Language Models Trained on Code (2021) — arXiv:2107.03374
  • Liu et al., Is Your Code Generated by ChatGPT Really Correct? (EvalPlus) (2023) — arXiv:2305.01210