Stats
Actions
Tags
From toolkit
Builds AI agent evaluations using Anthropic patterns: code/model/human graders, tasks, trials, benchmarks for coding, conversational, research agents.
How this skill is triggered — by the user, by Claude, or both
Slash command
/toolkit:anthropic-evaluationsThis skill is limited to the following tools:
The summary Claude sees in its skill listing — used to decide when to auto-load this skill
Build rigorous evaluations for AI agents using Anthropic's proven patterns.
Build rigorous evaluations for AI agents using Anthropic's proven patterns.
You MUST read the reference files for detailed guidance:
YAML Templates:
Annotated Examples:
| Term | Definition |
|---|---|
| Task | Single test with defined inputs and success criteria |
| Trial | One attempt at a task (run multiple for consistency) |
| Grader | Logic that scores agent performance; tasks can have multiple |
| Transcript | Complete record of a trial (outputs, tool calls, reasoning) |
| Outcome | Final state in environment (not just what agent said) |
| Evaluation harness | Infrastructure that runs evals end-to-end |
| Agent harness | System enabling model to act as agent (scaffold) |
| Evaluation suite | Collection of tasks measuring specific capabilities |
| Type | Methods | Best For |
|---|---|---|
| Code-based | String match, unit tests, static analysis, state checks | Fast, cheap, objective verification |
| Model-based | Rubric scoring, assertions, pairwise comparison | Nuanced, open-ended tasks |
| Human | SME review, A/B testing, spot-check sampling | Gold standard calibration |
See Grader Types for detailed comparison.
| Type | Question | Target Pass Rate |
|---|---|---|
| Capability | "What can this agent do well?" | Start low, hill-climb |
| Regression | "Does it still handle what it used to?" | Near 100% |
Capability evals with high pass rates "graduate" to regression suites.
| Metric | Measures | Use When |
|---|---|---|
| pass@k | At least 1 success in k attempts | One success matters (coding) |
| pass^k | All k attempts succeed | Consistency essential (customer-facing) |
Example: 75% per-trial success rate
tracked_metrics:
- type: transcript
metrics: [n_turns, n_toolcalls, n_total_tokens]
- type: latency
metrics: [time_to_first_token, output_tokens_per_sec, time_to_last_token]
Based on Demystifying evals for AI agents by Anthropic (January 2026).
npx claudepluginhub dwmkerr/claude-toolkit --plugin toolkitRuns evaluations on ADK agents: writing eval datasets, analyzing failures, comparing results, and optimizing agents using the Quality Flywheel methodology.
Builds evaluation frameworks for agent systems to test performance systematically, validate context engineering choices, and measure improvements over time.