Skill

agentv-eval-review

Lints AgentV eval YAML files for structural issues, schema compliance, and quality problems using a Python script, followed by LLM semantic review for factual accuracy, coverage, and consistency.

Python

testing

code-quality

npx claudepluginhub entityprocess/agentv --plugin agentv-dev

Tool Access

This skill uses the workspace's default tool permissions.

Preview

Lint and review AgentV eval YAML files for structural issues, schema compliance, and quality problems. Runs deterministic checks via script, then applies LLM judgment for semantic issues the script cannot catch.

Supporting Assets

scripts/lint_eval.py

SKILL.md

Similar Skills

agentv-eval-writer

Writes, edits, reviews, and validates AgentV EVAL.yaml files for agent skill evaluations. Adds test cases, configures graders, converts from evals.json or chat transcripts.

4 files

agentv-dev

agent-plugin-review

Reviews AI plugin PRs: structural linting with Python script, eval YAML checks, skill quality audits via checklist. Outputs inline PR comments.

3 files

agentic-engineering

skill-forge-eval

Runs evaluation pipelines on Claude Code skills to test triggering accuracy, workflow correctness, and output quality. Spawns sub-agents for parallel execution and generates JSON reports.

skill-forge

Stats

Parent Repo Stars11

Parent Repo Forks0

Last CommitMar 25, 2026

Actions

View Source View Plugin View on GitHub View README

Help us improve

Share bugs, ideas, or general feedback.

Eval Review

Overview

Process

Step 1: Run the linter

Execute scripts/lint_eval.py against the target eval files:

python scripts/lint_eval.py <path-to-evals-dir-or-file> --json

The script checks:

.eval.yaml extension

description field present

Each test has id, input, and at least one of criteria/expected_output/assertions

File paths in type: file use leading /

assertions blocks present (flags tests relying solely on expected_output)

expected_output prose detection (flags "The agent should..." patterns)

Repeated file inputs across tests (recommends top-level input)

Naming prefix consistency across eval files in same directory

Step 2: Review script output

Report the script findings grouped by severity (error > warning > info). For each finding, include the file path and a concrete fix.

Step 3: Semantic review (LLM judgment)

The script catches structural issues but cannot assess:

Factual accuracy — Do tool/command names in expected_output match what the skill documents?

Coverage gaps — Are important edge cases missing?

Assertion discriminability — Would assertions pass for both good and bad output?

Cross-file consistency — Do output filenames match across evals and skills?

Read the relevant SKILL.md files and cross-check against the eval content for these issues.

Skill Resources

scripts/lint_eval.py — Deterministic eval linter (Python 3.11+, stdlib only)