EXPLAINER

Common Evaluation Metrics (Accuracy, F1, BLEU, ROUGE, Perplexity)

conceptfreshness: stablesources not yet verified

Different tasks need different metrics: accuracy/precision/recall/F1 for classification, BLEU/ROUGE for overlap with references, perplexity for language modelling, and rubric- or judge-based scores for open-ended generation.

Short answer: Pick metrics that match the decision you care about. Accuracy misleads under class imbalance; BLEU/ROUGE reward word overlap, not truth; perplexity measures prediction of text, not usefulness. Combine automatic metrics with human or rubric-based checks.

What is it?

Scores are tools, not truth. Accuracy counts how often you are right, but is misleading when one outcome is rare. BLEU and ROUGE count word overlap with a reference answer. Perplexity says how surprised a model is by text. None of them alone says whether an answer is good.

Also known as: F1 score, precision and recall, BLEU score, ROUGE, perplexity, accuracy vs F1, ROC AUC, BERTScore.

Technical definition

Classification: accuracy, precision, recall, F1, ROC-AUC, calibration (expected calibration error). Generation with references: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019) — correlate imperfectly with human judgement, especially for open-ended outputs. Language modelling: perplexity/bits-per-byte. Code: pass@k. Retrieval: recall@k, MRR, nDCG. Open-ended: rubric scoring, LLM-as-a-Judge and human review. Report confidence intervals and compare under identical protocols.

Entity type: concept · Freshness class: stable.

When should I use it?

Pick metrics at the start of any project.

Implementation patterns

  • Choose metric from cost of errors (false positives vs negatives).
  • Report multiple metrics plus confidence intervals.
  • Use human review for generation quality.
  • Fix preprocessing and tokenisation for overlap metrics.

Technologies and frameworks

Commonly associated: scikit-learn metrics, sacreBLEU, rouge-score, evaluate (Hugging Face). Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. List error costs.
  2. Choose primary and secondary metrics.
  3. Compute with a standard library.
  4. Add uncertainty.
  5. Validate against human judgement.

Common failure modes

  • Accuracy on imbalanced data.
  • BLEU/ROUGE treated as quality.
  • Comparing perplexities across different tokenisers.
  • No uncertainty estimates.

Troubleshooting

  • High accuracy, useless model — Look at the confusion matrix and per-class recall.
  • Metrics disagree with reviewers — Adopt rubric or judge-based evaluation for that task.

Contrasted with

Questions this page answers

  • What is the F1 score?
  • What is perplexity?
  • What is the difference between precision and recall?
  • What is BLEU used for?
  • Which metric should I use for summarisation?

Typical goals: choose the right metric; interpret a reported score.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.

Canonical sources (unverified leads):

  • Papineni et al., BLEU: a Method for Automatic Evaluation of Machine Translation, ACL (2002) — locator to be identified
  • Lin, ROUGE: A Package for Automatic Evaluation of Summaries (2004) — locator to be identified
  • Zhang et al., BERTScore (2019) — arXiv:1904.09675