add-llm-evals

Add evaluations to an LLM app

Observability tells you what happened; evaluation tells you whether it was good. Add both an offline suite (runs in CI on a fixed dataset) and, optionally, online scoring (grades production traffic).

Pick the eval type per metric

Two families, use both where relevant:

• Reference-based (you have a ground-truth answer): exact/fuzzy match, semantic similarity, ROUGE/BLEU - cheap, deterministic. Good for classification, extraction, closed QA.
• Reference-free / LLM-as-a-judge (open-ended output): a strong model scores the output against a rubric (faithfulness, relevance, coherence, safety). Scalable proxy for human judgment - but calibrate it (see caveats).

For RAG specifically, use the standard quartet: faithfulness, answer relevance, context precision, context recall (Ragas implements these).

Build the offline suite (do this first)

1. Curate a dataset - 20-100 representative inputs with expected outputs or rubrics. Pull real cases from production traces (this is where observability + eval connect). Version it in the repo.
2. Choose a framework - promptfoo (YAML, great for prompt/RAG + CI), DeepEval (pytest-style, 40+ metrics), or Ragas (RAG metrics). Pick one; don't hand-roll.
3. Define pass criteria - per-metric thresholds (e.g. faithfulness ≥ 0.8). Fail the build if a threshold regresses.
4. Wire into CI - run on every PR that touches prompts/models/retrieval. Store scores so you can see trends, not just pass/fail.

See references/frameworks.md for a minimal promptfoo config and a DeepEval test example.

Add online evaluation (optional, higher value)

Sample production traffic and score it with LLM-as-a-judge (most observability platforms - Langfuse, Phoenix, Opik, Braintrust - run these on live traces). Alert when a quality score drops. This catches drift the offline suite can't (real inputs shift over time).

Calibrate LLM-as-a-judge (don't skip)

• Anchor to humans: label ~30 examples yourself, then check the judge agrees. Iterate the rubric until agreement is high.
• Watch known biases: position bias, verbosity bias, self-preference. Randomize order; keep rubrics concrete.
• Use a strong judge model and a low temperature; a weak judge produces noisy scores.

Verify

• Offline suite runs green in CI and fails when you intentionally break a prompt.
• Scores are stored/trended, not just printed.
• (If online) a dashboard shows quality over time with an alert threshold.

Anti-patterns

• Only "vibe-checking" outputs manually - doesn't scale, doesn't catch regressions.
• A dataset of toy inputs that don't resemble production.
• Trusting an uncalibrated LLM judge as ground truth.
• Evals that never fail the build (then they're decoration, not a gate).

Research grounding

LLM-as-a-judge: Zheng et al. 2023 (MT-Bench/Chatbot Arena). Reference-free hallucination detection: SelfCheckGPT (Manakul et al. 2023). See this repo's README → Research & Benchmarks.