Eval Skills for AI Coding Agents
Skills that guide AI coding agents to help you build LLM evaluations.
These skills guard against common mistakes I've seen helping 50+ companies and teaching students in our AI Evals course. If you're new to evals, see questions.md for free resources on the fundamentals.
New to Evals? Start Here
If you are new to evals, start with the eval-audit skill. Give your coding agent these instructions:
Install the eval skills plugin from https://github.com/hamelsmu/evals-skills, then run /evals-skills:eval-audit on my eval pipeline. Investigate each diagnostic area using a separate subagent in parallel, then synthesize the findings into a single report. Use other skills in the plugin as recommended by the audit.
The audit isn't a complete solution, but it will catch common problems we've seen in evals. It will also recommend other skills to use to fix the problems.
Installation
In Claude Code, run these two commands:
# Step 1: Register the plugin repository
/plugin marketplace add hamelsmu/evals-skills
# Step 2: Install the plugin
/plugin install evals-skills@hamelsmu-evals-skills
To upgrade:
/plugin update evals-skills@hamelsmu-evals-skills
After installation, restart Claude Code. The skills will appear as /evals-skills:<skill-name>.
Installation (npx skills)
If you use the open Skills CLI, install from this repo with:
npx skills add https://github.com/hamelsmu/evals-skills
Install one skill only:
npx skills add https://github.com/hamelsmu/evals-skills --skill eval-audit
Check for updates:
npx skills check
npx skills update
Available Skills
| Skill | What it does |
|---|
| eval-audit | Audit an eval pipeline and surface problems with prioritized severity |
| error-analysis | Guide the user through reading traces and categorizing failures |
| generate-synthetic-data | Create diverse synthetic test inputs using dimension-based tuple generation |
| write-judge-prompt | Design LLM-as-Judge evaluators for subjective quality criteria |
| validate-evaluator | Calibrate LLM judges against human labels using data splits, TPR/TNR, and bias correction |
| evaluate-rag | Evaluate retrieval and generation quality in RAG pipelines |
| build-review-interface | Build custom annotation interfaces for human trace review |
Invoke a skill with /evals-skills:skill-name, e.g., /evals-skills:error-analysis.
Write Your Own Skills
These skills are a starting point and only encode common mistakes that generalize across projects. Skills grounded in your stack, your domain, and your data will outperform them. Start here, then write your own.
The meta-skill can help you ground custom skills.
Beyond These Skills
These skills handle the parts of eval work that generalize across projects. Much of the process doesn't: production monitoring, CI/CD integration, data analysis, and much more. The course covers all of it.
hamelsmu