Inspect (UK AI Security Institute Evaluation Framework)

What is it?

Think of Inspect as a test harness for AI models. You write down questions (the dataset), how the model should attempt them (the solver — a plain prompt, a tool-using agent, a multi-step plan), and how to mark the answers (the scorer — exact match, a model acting as a grader, or custom code). Inspect runs it against whichever model you point it at, keeps detailed logs, and gives you a viewer to see exactly what happened.

Also known as: Inspect AI, inspect_ai, UK AISI Inspect, AI Security Institute Inspect, Inspect evals, inspect_evals, Inspect framework, UK AI Safety Institute evaluation framework.

Technical definition

Inspect's README describes it as a framework for large language model evaluations created by the UK AI Security Institute, with built-in components for prompt engineering, tool usage, multi-turn dialogue and model-graded evaluations, extensible through other Python packages, and a collection of over 200 pre-built evaluations. Its documentation organises the framework into: Tasks (datasets, solvers and scorers brought together), Datasets, Solvers (prompt engineering and other elicitation strategies), Scorers (including text-matching, multiple-choice, model-graded and math scorers, with metrics and epoch reduction); Models (a uniform API across providers, plus caching, concurrency control, batch mode, reasoning-model options and structured output); Agents (ReAct agent, a "deep agent", multi-agent composition, custom agents, an agent bridge for frameworks such as OpenAI Agents SDK, LangChain and Pydantic AI, and a human agent for baselining); Tools (custom tools, standard tools for code execution, text editing, computer use, web search and browsing, and MCP tools); Sandboxing to isolate model-generated code; tool approval and result-review policies; limits on time, messages, tokens and cost; eval sets for running many tasks; log files, dataframes and a log viewer ("Inspect View") with a VS Code extension; and scanners that review transcripts for problems such as misconfigured environments or refusals. The pre-built evaluations live in the separate Inspect Evals repository, which is maintained by Generality Labs with contributions from the UK AI Security Institute and others, not by the Institute alone. The source repository licence is MIT.

Entity type: framework · Freshness class: volatile.

When should I use it?

Use when you need repeatable, inspectable LLM or agent evaluations, particularly with tools and sandboxes. For a quick check of one prompt, a handful of manual test cases may be enough.

Implementation patterns

Define a Task from a dataset, a solver and a scorer; run it with the inspect CLI against a model identifier.
Use model-graded scorers for open-ended answers, and spot-check them against human labels.
Run agent evaluations inside a sandbox with limits on tokens, messages and time.
Re-score saved logs without re-running the model.
Run eval sets across several models and compare with the analysis tools.

Technologies and frameworks

Commonly associated: inspect_ai (Python), Inspect Evals, Inspect View, VS Code extension, Docker-based sandboxes, model providers via a uniform API. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

Write 20–50 representative samples with clear targets.
Choose the simplest solver that reflects real use (plain generation first, tools later).
Pick a scorer you can audit; add a model-graded scorer only when exact checks are impossible.
Run against a small model first to debug, then your target models.
Open the log viewer and read failures before reading the aggregate score.
Record versions and settings so others can reproduce the run.

Common failure modes

Treating a pre-built eval score as proof of real-world safety or capability (Public Benchmarks vs Task-Specific Evals).
Letting an agent run unsandboxed code with real credentials.
Trusting a model-graded score without calibration against humans (LLM-as-a-Judge).
Comparing scores across runs with different prompts, epochs or limits.
Evaluating on data the model may have seen (Benchmark Contamination and Data Leakage).

Troubleshooting

Scores differ between runs — Fix seeds where supported, set the number of epochs, and report uncertainty rather than a single number.
Agent evaluation hangs or costs too much — Set message, token, time and cost limits on the task.
Cannot reproduce a published result — Check the eval version, prompt, model snapshot and scorer settings recorded in the log.

Contrasted with

MLflow — MLflow tracks experiments and models across the ML lifecycle; Inspect specialises in LLM evaluation harness features (solvers, scorers, sandboxes).
LLM-as-a-Judge — Model grading is one scorer type inside Inspect, not a replacement for the framework.
HumanEval and Code Generation Benchmarks — HumanEval is a benchmark; Inspect is the harness that can run benchmarks like it.

Questions this page answers

What is Inspect from the UK AI Security Institute?
How do I write an evaluation in Inspect?
What are solvers and scorers in Inspect?
How does Inspect sandbox agent tools?
Where do I find ready-made Inspect evaluations?

Typical goals: run standard LLM evaluations reproducibly; build a custom evaluation with scoring and logs; evaluate an agent with sandboxed tools.

Sources and verification

Status: key claims checked on 2026-10-09 against the primary sources listed below; everything else is not independently verified. The claims under "Checked against primary sources" were compared with the linked pages on that date. Pages change, so re-check anything time-sensitive before relying on it.

Checked against primary sources (2026-10-09):

Inspect is a framework for LLM evaluations created by the UK AI Security Institute; built-in components cover prompt engineering, tool use, multi-turn dialogue and model-graded evaluations; extensible through Python packages.
README states over 200 pre-built evaluations; the repository licence text is MIT, copyright UK AI Security Institute.
Documentation structure: Tasks combine datasets, solvers and scorers; agents (ReAct, deep agent, multi-agent, agent bridge, human agent); tools (standard tools including code execution, computer use, web search and browsing; MCP tools); sandboxing; tool approval; limits; eval sets; logs, dataframes, viewer, VS Code extension; scanners.
The Inspect Evals repository says it is maintained by Generality Labs, founded in 2024 with contributions from the UK AI Security Institute, Arcadia Impact and the Vector Institute.

Primary sources consulted:

Not independently verified:

Exact import paths, function names and CLI flags: this page deliberately includes no code sketch because the tutorial was not read in full.
The PyPI package name and current version.
The number of evaluations today.
The Institute's name history (the organisation was previously called the AI Safety Institute); not checked.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

Component names and APIs (the project is actively developed).
The number and list of pre-built evaluations ("over 200" per the README on the check date).
Supported providers and sandbox types.
Maintenance and ownership of Inspect Evals.

Further reading and canonical sources (not all checked):

Inspect documentation — inspect.aisi.org.uk
Inspect repository (UKGovernmentBEIS/inspect_ai) — github.com/UKGovernmentBEIS/inspect_ai
Inspect Evals repository and docs — github.com/UKGovernmentBEIS/inspect_evals