EXPLAINER

LLM-as-a-Judge

techniquefreshness: evolvingsources not yet verified

LLM-as-a-judge uses a language model to grade other model outputs against a rubric or compare pairs. It scales evaluation but has biases and must be calibrated against humans.

Short answer: A strong model scores or ranks outputs using a rubric. It is cheap and fast, but exhibits position, verbosity and self-preference biases and can be wrong; calibrate against human labels and use clear rubrics, pairwise comparisons and randomised order.

What is it?

Instead of paying people to read thousands of answers, you ask an AI to grade them using clear criteria. It is quick and consistent, but the AI judge has its own quirks — like favouring longer answers — so you need to check it against human judgement.

Also known as: LLM judge, model graded evaluation, AI judge, GPT-as-judge, automatic evaluation with LLM, pairwise comparison judge, rubric grading LLM, MT-Bench.

Technical definition

Zheng et al. (2023, MT-Bench/Chatbot Arena) report that strong LLM judges can agree with human preferences at rates comparable to human–human agreement on some settings, while documenting position bias, verbosity bias and self-enhancement bias, and limited maths/reasoning grading ability. Practices: pairwise or reference-guided grading, rubric prompts with explicit criteria, order randomisation, multiple judges/samples, chain-of-thought justifications that are logged (not necessarily shown to users), and measured agreement with a human-labelled subset. Judge models and prompts must be versioned like code.

Entity type: technique · Freshness class: evolving.

When should I use it?

Use for open-ended tasks where exact-match scoring is impossible, with human calibration.

Implementation patterns

  • Pairwise comparison with randomised A/B order.
  • Rubric with 3–5 specific criteria and anchors.
  • Reference-guided grading when ground truth exists.
  • Periodic human audit of a sample; track judge–human agreement.
  • Use a different (or stronger) model family as judge to reduce self-preference.

Technologies and frameworks

Commonly associated: MT-Bench, G-Eval-style prompts, RAGAS, evaluation platforms with judge support. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Define criteria and scoring anchors.
  2. Collect 50–100 human-labelled examples.
  3. Measure judge–human agreement.
  4. Iterate on the rubric.
  5. Monitor drift and audit regularly.

Common failure modes

  • Position and length bias.
  • Judge shares the evaluated model's blind spots.
  • Vague rubrics giving noisy scores.
  • Judge errors on factual or mathematical correctness.
  • Silent changes when the judge model is updated.

Troubleshooting

  • Scores drift over time — Pin judge model version and prompt; re-calibrate on a human set.
  • Judge prefers longer answers — Add length-controlled comparisons or penalise padding explicitly.

Contrasted with

Questions this page answers

  • What is LLM-as-a-judge?
  • Can I trust an LLM to grade answers?
  • What biases do LLM judges have?
  • How do I write a good judging rubric?
  • How do I validate an LLM judge?

Typical goals: scale up evaluation of open-ended outputs; calibrate an automated grader.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — arXiv:2306.05685
  • Liu et al., G-Eval (2023) — arXiv:2303.16634
Learning path: Evaluating AI systems (step 5 of 7)
← Public Benchmarks vs Task-Specific EvalsRAG Evaluation (Retrieval and Answer Quality) →