Stats
Actions
Tags
'%20stop-opacity%3D'0.16'%2F%3E%3Cstop%20offset%3D'1'%20stop-color%3D'rgb(200%2C90%2C60)'%20stop-opacity%3D'0.03'%2F%3E%3C%2FlinearGradient%3E%3C%2Fdefs%3E%3Crect%20width%3D'320'%20height%3D'200'%20fill%3D'url(%23g)'%2F%3E%3Ccircle%20cx%3D'250'%20cy%3D'56'%20r%3D'92'%20fill%3D'rgb(200%2C90%2C60)'%20fill-opacity%3D'0.06'%2F%3E%3Ccircle%20cx%3D'64'%20cy%3D'172'%20r%3D'58'%20fill%3D'rgb(200%2C90%2C60)'%20fill-opacity%3D'0.05'%2F%3E%3C%2Fsvg%3E)
From Napkin
Build and run evaluators for AI/LLM applications using Phoenix. Covers code-first eval design, judge model selection, RAG evals, synthetic data generation, and production guardrails.
How this skill is triggered — by the user, by Claude, or both
Slash command
/napkin:phoenix-evalsThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
references/axial-coding.mdreferences/common-mistakes-python.mdreferences/error-analysis-multi-turn.mdreferences/error-analysis.mdreferences/evaluate-dataframe-python.mdreferences/evaluators-code-python.mdreferences/evaluators-code-typescript.mdreferences/evaluators-custom-templates.mdreferences/evaluators-llm-python.mdreferences/evaluators-llm-typescript.mdreferences/evaluators-overview.mdreferences/evaluators-pre-built.mdreferences/evaluators-rag.mdreferences/experiments-datasets-python.mdreferences/experiments-datasets-typescript.mdreferences/experiments-overview.mdreferences/experiments-running-python.mdreferences/experiments-running-typescript.mdreferences/experiments-synthetic-python.mdreferences/experiments-synthetic-typescript.mdBuild evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.
Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview
Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}
RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)
Production: production-overview → production-guardrails → production-continuous
| Prefix | Description |
|---|---|
fundamentals-* | Types, scores, anti-patterns |
observe-* | Tracing, sampling |
error-analysis-* | Finding failures |
axial-coding-* | Categorizing failures |
evaluators-* | Code, LLM, RAG evaluators |
experiments-* | Datasets, running experiments |
validation-* | Validating evaluator accuracy against human labels |
production-* | CI/CD, monitoring |
| Principle | Action |
|---|---|
| Error analysis first | Can't automate what you haven't observed |
| Custom > generic | Build from your failures |
| Code first | Deterministic before LLM |
| Validate judges | >80% TPR/TNR |
| Binary > Likert | Pass/fail, not 1-5 |
npx claudepluginhub ani1797/forge --plugin copilot-sdk3plugins reuse this skill
First indexed Jun 6, 2026
Build and run evaluators for AI/LLM applications using Phoenix. Covers code-first eval design, judge model selection, RAG evals, synthetic data generation, and production guardrails.
Builds LangSmith evaluation pipelines: create LLM-as-Judge/custom evaluators, capture agent outputs/trajectories via run functions, run locally with evaluate() or CLI.
Manages LLM-as-judge evaluation workflows on Arize: creates/updates evaluators, runs evaluations on spans/experiments, manages tasks, trigger-runs, column mapping, continuous monitoring.