cli-agent-evals

CLI Agent Evals

Unified live eval toolkit for coding-agent CLIs.

cli-evals runs the same scenario suite across Claude Code, OpenAI Codex, opencode, and GitHub Copilot by driving real interactive CLIs through @agent-ix/agent-pty.

Install

pnpm add -D @agent-ix/cli-agent-evals

Or install globally:

npm install -g @agent-ix/cli-agent-evals

Prerequisites for live runs:

• tmux >= 3.0
• claude, codex, opencode, or Copilot/gh on PATH
• authenticated local agent sessions for the agents you run

Check local readiness:

cli-evals doctor

Quickstart

Create cli-agent-evals.config.mjs:

import { defineSuite } from "@agent-ix/cli-agent-evals";

export default defineSuite({
  name: "my-cli",
  rootDir: import.meta.dirname,
  scenarios: [
    {
      id: "EV-001",
      canary: true,
      prompt:
        "Use my CLI to complete the happy path, then print the success sentinel.",
    },
  ],
});

Run canaries:

cli-evals run --suite ./cli-agent-evals.config.mjs --canary --agent claude --model sonnet

Run one scenario and keep the workdir:

cli-evals run --suite ./cli-agent-evals.config.mjs --filter EV-001 --agent codex --model gpt-5 --keep

Print a previous report:

cli-evals rebuild --report evals/reports/latest.json

Plugin Install

Claude Code:

claude plugin marketplace add agent-ix/cli-agent-evals
claude plugin install cli-agent-evals

Codex:

codex plugin marketplace add agent-ix/cli-agent-evals
codex plugin add cli-agent-evals

opencode:

gh skill install agent-ix/cli-agent-evals --all --scope user --agent opencode

GitHub Copilot:

gh skill install agent-ix/cli-agent-evals --all --scope user --agent github-copilot

Installed skills:

• cli-evals: run, inspect, and debug suites.
• cli-evals-author: draft scenarios, fixtures, assertions, and classifiers.

Docs

• Usage
• Suite authoring
• Agent drivers
• Reports
• Plugin install