What is it?
When a RAG answer is wrong, either the system fetched the wrong material or the model misused the right material. Testing the two parts separately tells you which to fix.
Also known as: evaluate RAG, RAG metrics, RAGAS, faithfulness metric, groundedness, recall@k, context precision, context recall.
Technical definition
Retrieval metrics require relevance labels (which chunks answer each question): recall@k, precision@k, MRR, nDCG. Generation metrics include faithfulness (are claims supported by retrieved context?), answer relevance, context utilisation and abstention accuracy; automated frameworks such as RAGAS (Es et al., 2023) and ARES (Saad-Falcon et al., 2023) use LLM judges (LLM-as-a-Judge) and need calibration. End-to-end correctness against gold answers complements component metrics. Include unanswerable and adversarial questions.
Entity type: technique · Freshness class: evolving.
When should I use it?
Run before launch and on every pipeline change.
Implementation patterns
- Build 50–200 questions with gold passages and answers.
- Compute retrieval metrics before generation.
- Judge faithfulness against retrieved context.
- Add unanswerable questions to test abstention.
- Track metrics per change.
Technologies and frameworks
Commonly associated: RAGAS, ARES, TruLens-style tooling, custom harnesses. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Collect realistic questions.
- Label relevant chunks.
- Score retrieval.
- Score faithfulness and correctness.
- Segment failures and fix the largest bucket.
Common failure modes
- Only end-to-end scoring, hiding the failing component.
- Uncalibrated LLM judges.
- Test questions written from the chunks (too easy).
- No negative/unanswerable cases.
Troubleshooting
- Good retrieval, bad answers — Tighten the prompt, reduce context noise, require citations.
- Bad retrieval — Revisit chunking, embeddings, hybrid search and reranking.
Contrasted with
- How to Evaluate AI Systems and Outputs — General evaluation principles; this entry applies them to retrieval-augmented systems.
Questions this page answers
- How do I evaluate a RAG system?
- What is faithfulness in RAG?
- What is recall@k?
- What metrics should I use for RAG?
- How do I build a RAG test set?
Typical goals: measure and improve RAG quality; find whether retrieval or generation is the problem.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.
Canonical sources (unverified leads):
- Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation (2023) — arXiv:2309.15217
- Saad-Falcon et al., ARES (2023) — arXiv:2311.09476
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — arXiv:2005.11401
Try it with GoVibeTool
For a manual groundedness spot-check on a single answer, the Fact Anchor Checker compares each claim against source text you paste and marks what is supported, partially supported or not found. It is a review aid for individual answers, not an automated RAG metric.
Runs in your browser; the text you paste is not sent to a server.