What is it?
Show people two answers without saying which model wrote them, ask which they prefer, and repeat thousands of times. The votes produce a ranking. It captures what people like — which may include confident tone and formatting — not strictly what is true.
Also known as: chatbot arena, LMArena, Elo rating LLM, Bradley-Terry, pairwise preference votes, human evaluation of LLMs, arena leaderboard, A/B testing models.
Technical definition
Chatbot Arena (Chiang et al., 2024) crowdsources pairwise comparisons and fits Bradley–Terry style ratings with confidence intervals; related work studies style-control and prompt-difficulty adjustments. Biases include verbosity and formatting preference, population skew, prompt distribution and gaming risks. Private human evaluation uses the same idea with controlled raters and rubrics. The best-known public arena has been renamed more than once — Chatbot Arena, then LMArena, and since January 2026 "Arena" at arena.ai (name and ownership are time-sensitive). Leaderboard specifics (methodology, style control, categories) change and should be read from the current source.
Entity type: technique · Freshness class: volatile.
When should I use it?
Use for subjective quality (helpfulness, tone) where correctness metrics do not capture preference.
Implementation patterns
- Blind pairwise comparison with randomised order.
- Stratify by task type and difficulty.
- Report confidence intervals and sample sizes.
- Combine with correctness-based evals.
Technologies and frameworks
Commonly associated: Chatbot Arena / LMArena, Bradley–Terry models, annotation platforms. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define the comparison task.
- Recruit appropriate raters.
- Blind and randomise.
- Fit ratings with uncertainty.
- Cross-check with objective evals.
Common failure modes
- Style over substance rewarded.
- Unrepresentative voters or prompts.
- Ties and noise handled poorly.
- Treating small rating differences as meaningful.
Troubleshooting
- Rankings seem to favour verbose models — Use style-controlled views or length-matched comparisons.
- Your sample is small — Report intervals; gather more votes or restrict claims.
Contrasted with
- LLM-as-a-Judge — LLM judges approximate human preferences automatically.
- MMLU (Massive Multitask Language Understanding) — Fixed-answer benchmarks score correctness; arenas score preference.
Questions this page answers
- What is Chatbot Arena?
- How are LLM Elo ratings calculated?
- Are human preference leaderboards reliable?
- How do I run a human evaluation of my chatbot?
Typical goals: understand how arena rankings are made; run your own human evaluation.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.
Time-sensitive claims to verify before relying on them:
- Current leaderboard methodology, rankings and name of the platform.
Canonical sources (unverified leads):
- Chiang et al., Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024) — arXiv:2403.04132
- Zheng et al., Judging LLM-as-a-Judge (2023) — arXiv:2306.05685