Skill

benchmark

Run benchmark suites and manage policy evolution — create challengers, compare against champions, promote or rollback policies.

npx claudepluginhub hmbown/plugins --plugin autodialectics

Tool Access

This skill uses the workspace's default tool permissions.

Preview

Use this skill to benchmark policies and drive champion/challenger evolution.

SKILL.md

Similar Skills

cache-components

139.4k

Guides Next.js Cache Components and Partial Prerendering (PPR): 'use cache' directives, cacheLife(), cacheTag(), revalidateTag() for caching, invalidation, static/dynamic optimization. Auto-activates on cacheComponents: true.

cache-components

pdf

131.6k

Processes PDFs: extracts text/tables/images, merges/splits/rotates pages, adds watermarks, creates/fills forms, encrypts/decrypts, OCRs scans. Activates on PDF mentions or output requests.

11 files

document-skills

Stats

Parent Repo Stars0

Parent Repo Forks0

Last CommitApr 12, 2026

Actions

View Source View Plugin View on GitHub View README

Help us improve

Share bugs, ideas, or general feedback.

Autodialectics Benchmark & Evolve

Use this skill to benchmark policies and drive champion/challenger evolution.

Prerequisites

autodialectics-mcp must be on PATH (pip install autodialectics).

MCP Workflow

Benchmarking

benchmark(suite_dir?, policy_id?) — run the benchmark suite against a policy. Returns case-by-case results with scores and decisions.

Policy Evolution

evolve_policy(use_gepa?) — analyze recent benchmark reports and create a challenger policy. Set use_gepa: false to skip the GEPA optimizer (simpler heuristic fallback).

promote_policy(policy_id) — promote a challenger to champion if comparison rules allow.

rollback_policy() — revert to the previous champion if the current one regresses.

CLI Fallback

autodialectics benchmark autodialectics evolve autodialectics promote <policy_id> autodialectics rollback

Typical Evolution Cycle

benchmark → evolve_policy → benchmark (with challenger) → compare → promote or rollback

Run benchmarks with the current champion to establish a baseline.

Evolve a challenger from the benchmark reports.

Run the same benchmarks with the challenger's policy ID.

Compare results. Promote if the challenger wins; rollback if it regresses.

Guidance

Never claim a policy is better without benchmark evidence from the same suite.

When reporting benchmark results, include: total cases, pass/fail/revise counts, mean overall score, mean slop composite.

If evolve_policy returns no_reports, run benchmarks first to generate data.

Promotion can be denied by comparison rules — check the response status.

Arguments

If the user passes a suite directory after /autodialectics:benchmark, use it as the benchmark suite path.

Autodialectics Benchmark & Evolve

Use this skill to benchmark policies and drive champion/challenger evolution.

Prerequisites

autodialectics-mcp must be on PATH (pip install autodialectics).

MCP Workflow

Benchmarking

benchmark(suite_dir?, policy_id?) — run the benchmark suite against a policy. Returns case-by-case results with scores and decisions.

Policy Evolution

evolve_policy(use_gepa?) — analyze recent benchmark reports and create a challenger policy. Set use_gepa: false to skip the GEPA optimizer (simpler heuristic fallback).
promote_policy(policy_id) — promote a challenger to champion if comparison rules allow.
rollback_policy() — revert to the previous champion if the current one regresses.

CLI Fallback

autodialectics benchmark
autodialectics evolve
autodialectics promote <policy_id>
autodialectics rollback

Typical Evolution Cycle

benchmark → evolve_policy → benchmark (with challenger) → compare → promote or rollback

Run benchmarks with the current champion to establish a baseline.
Evolve a challenger from the benchmark reports.
Run the same benchmarks with the challenger's policy ID.
Compare results. Promote if the challenger wins; rollback if it regresses.

Guidance

Never claim a policy is better without benchmark evidence from the same suite.
When reporting benchmark results, include: total cases, pass/fail/revise counts, mean overall score, mean slop composite.
If evolve_policy returns no_reports, run benchmarks first to generate data.
Promotion can be denied by comparison rules — check the response status.

Arguments

If the user passes a suite directory after /autodialectics:benchmark, use it as the benchmark suite path.