EXPLAINER

SWE-bench (Software Engineering Benchmark)

benchmarkfreshness: volatilesources not yet verified

SWE-bench evaluates whether AI systems can resolve real GitHub issues by editing a repository so that tests pass. It is a common measure of coding agents, with several variants and known caveats.

Short answer: SWE-bench gives an agent a real repository and issue, and checks whether its patch makes the failing tests pass without breaking others. Scores depend on the scaffold/agent, subset (e.g. Verified) and test quality — compare like for like.

What is it?

SWE-bench asks AI to act like a junior developer: here is a real bug report from an open-source project — fix the code. The test is whether the project's tests pass afterwards. It is far closer to real work than writing a single function.

Also known as: SWE-bench, SWE bench Verified, software engineering benchmark, coding agents benchmark, GitHub issues benchmark, SWE-agent, resolve rate.

Technical definition

Jimenez et al. (2023) built tasks from real issues and pull requests in popular Python repositories. Variants include human-validated subsets (Verified), lighter and multimodal sets, and multilingual extensions. Results depend heavily on the agent scaffold (tools, retrieval, loop), compute budget, and handling of flawed or under-specified tests; contamination and overfitting to the public set are concerns (Benchmark Contamination and Data Leakage). Leaderboards and subset definitions change.

Entity type: benchmark · Freshness class: volatile.

When should I use it?

Use to compare coding agents on realistic issue resolution.

Implementation patterns

  • Build an in-house SWE-bench-style set from your own closed issues.
  • Report scaffold, budget and subset with every score.
  • Review passing patches for quality, not just tests.

Technologies and frameworks

Commonly associated: SWE-agent, Docker evaluation harness, agent scaffolds. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Define subset and harness version.
  2. Fix scaffold and budget.
  3. Run multiple seeds.
  4. Inspect failures.
  5. Build a private variant for your codebase.

Common failure modes

  • Comparing scores across different scaffolds or subsets.
  • Tests that accept incorrect patches.
  • Ignoring cost per resolved issue.
  • Public-set overfitting.

Troubleshooting

  • Passing patch is poor quality — Add review criteria; evaluate maintainability, not only test pass.
  • Scores vary run to run — Run multiple seeds and report variance.

Contrasted with

Questions this page answers

  • What is SWE-bench?
  • What is SWE-bench Verified?
  • How reliable are SWE-bench scores?
  • How do I evaluate an AI coding agent on my repo?

Typical goals: interpret coding-agent scores; evaluate an AI developer tool.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Current leaderboard results, subsets and evaluation rules.
  • Status of Verified/other variants.

Canonical sources (unverified leads):

  • Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023) — arXiv:2310.06770
  • SWE-bench project site and leaderboard — swebench.com (verify)