How to Evaluate AI Systems and Outputs

What is it?

Evaluation ("evals") is the discipline of measuring how well an AI system performs on the tasks you care about. Because generative output is open-ended and non-deterministic, a handful of impressive examples tells you very little. Evals turn "seems good" into evidence you can track over time.

Public benchmarks compare models on standard tasks, but they may not reflect your use case. The evaluation that matters most is the one built from your data and your definition of success.

Why does it matter?

Without evals you cannot tell whether a new prompt, model, chunking strategy or tool helped or quietly broke something. Evals let you:

• catch regressions when you change anything;
• compare options (prompt A vs. B, model X vs. Y) on facts;
• quantify risk — how often does it hallucinate or refuse wrongly?
• decide when it is good enough to ship.

How does it work?

1. Define success. For each task choose measurable properties: correctness, completeness, groundedness in sources, format validity, tone, safety, latency, cost.
2. Build a test set. Start with 30–100 realistic cases drawn from real usage, including hard and adversarial ones and cases where the right answer is "not found". Record expected outcomes or a rubric. Keep some cases unseen while you tune, so you do not overfit to them.
3. Choose graders.

- Code-based checks: exact match, schema validation, "does the quoted text appear in the source?", unit tests for generated code. Cheapest and most reliable where applicable. - Human review with a clear rubric: the gold standard for subjective quality, but slow. - Model-based judges ("LLM-as-judge"): a model scores outputs against a rubric. Scalable, but can be biased, inconsistent and wrong, so calibrate against human ratings before trusting it.

1. Run and record results for every change; compare to the baseline.
2. Inspect failures, not just the score. Categorise them and fix the cause.
3. Monitor in production with sampling and feedback, and add real failures to the test set.

For What is RAG (Retrieval-Augmented Generation)?, evaluate retrieval (was the right chunk in the top-k?) separately from generation (was the answer faithful to the chunks?). For What are AI Agents?, also evaluate the trajectory: tool choices, arguments, and whether the final state is correct.

Example

A minimal eval in code for a question-answering bot:

const cases = [
  { q: "Refund window for annual plans?", expectContains: "30 days", mustCite: "refund-policy" },
  { q: "Do you support fax?",            expectContains: "not found" }
];
let pass = 0;
for (const c of cases) {
  const out = await answer(c.q);              // your system
  const ok = out.text.toLowerCase().includes(c.expectContains)
          && (!c.mustCite || out.citations.includes(c.mustCite));
  if (ok) pass++; else console.log("FAIL:", c.q, out.text);
}
console.log(pass + "/" + cases.length);

Crude, but it already catches regressions and the failure list tells you what to fix.

When should I use it?

As early as you have a prompt you intend to reuse. The effort scales with the stakes: a personal helper needs a few spot checks; a customer-facing or decision-support system needs a maintained test set, rubric, human review and monitoring.

Common mistakes

• Evaluating on the cases you tuned on.
• Relying on one aggregate score that hides serious failures in important categories.
• Trusting an LLM judge without calibration.
• Testing only happy paths — include missing-information, ambiguous and hostile inputs.
• Treating public benchmark rank as proof of fitness for your task.
• Never refreshing the test set as usage changes.

Related concepts