evaluation-metrics

Evaluation Metrics

Overview

Use this parent skill when the main RAG problem is measuring quality, detecting regressions, or proving that a change actually helped. Route to the child skill that matches whether the failure is in retrieval, in generation, or in the evaluation harness itself.

Problem Statement

Teams ship RAG changes with no baseline, tune on vibes, and cannot tell whether a new chunker, embedding model, or prompt made things better or worse. Without metrics, retrieval and generation regressions ship silently and are only caught by users.

Key Concepts

• Retrieval metrics: Whether the right chunks were fetched (recall@k, precision@k, MRR, nDCG, context precision/recall).
• Generation metrics: Whether the answer is grounded and useful (faithfulness, answer relevancy, correctness).
• Golden set: A labeled question-and-answer set used as the fixed baseline for comparison.
• Regression harness: An automated evaluation run on every change, wired into CI.

Implementation Guide

Step 1: Isolate the Layer That Fails

Measure retrieval and generation separately. Bad answers from good context are a generation problem; good answers are impossible from bad context.

Step 2: Pick Metrics for That Layer

Use retrieval metrics when the right chunks are missing or mis-ranked, and generation metrics when context is good but answers are wrong, unsupported, or off-topic.

Step 3: Automate as a Regression Gate

Build a golden set, run the metrics on every change, and block merges that regress the baseline instead of evaluating manually.

When to Use This Skill

• Choosing which metrics to track for a new or existing RAG pipeline
• Proving whether a chunking, embedding, or prompt change helped
• Setting up an automated evaluation gate before deployment

When NOT to Use This Skill

• The pipeline is not yet built and there is nothing to measure
• The bottleneck is a known implementation bug, not quality measurement
• You already have a metrics suite and the problem is retrieval or generation design

Related Skills

• Retrieval Evaluation Metrics
• Generation Evaluation Metrics
• RAG Evaluation Frameworks
• Self-RAG

Metrics & Success Criteria

• Every RAG change is compared against a fixed golden-set baseline
• Retrieval and generation quality are tracked as separate numbers
• Regressions are caught in CI instead of by users