ciagent

CIAgent

Your eval score is stable. Your system isn't.

CIAgent tells you whether your agent's evals are lying to you:

• Run the suite N times — see which verdicts flip, with the blame attached (agent-variance → fix the agent, judge-flake → fix the eval)
• Audit your LLM judge against ground truth you already have
• Replace judge calls with deterministic fact-checks mined from your knowledge base

Built from a real study: 1 in 7 answers an LLM judge passed were wrong, and deterministic checks caught 8 of 8 — methodology and raw numbers in STUDY.md. Native adapters for OpenAI, Anthropic, and LangGraph; imports exported traces from any stack — OpenTelemetry (openllmetry, Google ADK), Langfuse, and LangSmith — verified against real captures from OpenAI, Anthropic, CrewAI, Google ADK, and the Claude Agent SDK. Runs inside pytest.

See It in 30 Seconds

No install, no API keys, no config — one command runs a bundled demo suite three times on synthetic traces:

uvx ciagent test --mock --runs 3

Run 1/3: 7/8 passed
Run 2/3: 7/8 passed
Run 3/3: 7/8 passed

────────────────────────────────────────────────────────────
Stability Report
────────────────────────────────────────────────────────────
Suite score across 3 runs: 88%  /  88%  /  88%     ← looks stable

⚠️  FLAKY — 3/8 queries flipped verdicts across runs:      ← is not
   "What's your return window?"    ❌✅✅  pass_rate=0.67  source: agent-variance
   "Do you ship internationally?"  ✅❌✅  pass_rate=0.67  source: agent-variance
   "How do I reset my password?"   ✅✅❌  pass_rate=0.67  source: agent-variance

   Flip sources: 3 agent-variance (fix the agent) │ 0 judge-flake (fix the eval)

The aggregate score is identical every run. Three of the eight verdicts underneath it flipped. A single-run eval score would never tell you — the stability report does, and attributes every flip to the layer that caused it. (The demo simulates a flaky agent; point it at your own with the spec below.)

A stable score is not a stable system

Run the identical eval three times and you can get 96% / 95% / 96% — rock solid — while individual queries flip verdicts every run. The aggregate holds because the errors move around. --runs N shows what a single run can't:

ciagent test --runs 3

Run 1/3: 18/19 passed
Run 2/3: 18/19 passed
Run 3/3: 18/19 passed

────────────────────────────────────────────────────────────
Stability Report
────────────────────────────────────────────────────────────
Suite score across 3 runs: 95%  /  95%  /  95%

⚠️  FLAKY — 2/19 queries flipped verdicts across runs:
   "What's your return window?"    ✅❌✅  pass_rate=0.67  source: agent-variance (answer changed)
   "Do you sell gift cards?"       ❌✅✅  pass_rate=0.67  source: judge-flake (same answer, verdict flipped)

   Flip sources: 1 agent-variance (fix the agent) │ 1 judge-flake (fix the eval) │ 0 infra-error (retry) │ 0 mixed

Stability verdict: FLAKY

Every flip is attributed to its source, so it's a routed work item, not a scary number: agent-variance means the agent produced different output (fix the prompt, retrieval, or temperature); judge-flake means the output — or every deterministic check's outcome — was identical but the LLM judge changed its mind (fix the rubric, or replace the judge with a deterministic check); infra-error means a judge API call failed (retry, fix nothing). Attribution is structural, not guessed: deterministic checks cannot flip on identical output, and per-layer sub-verdicts are compared across runs. The console shows observed facts; pass@k/pass^k estimates live in the JSON output, labeled as estimates.

Flaky-but-passing exits 0 so adoption won't break your CI; add --fail-on-flaky when you're ready to gate on it. Try it with zero API keys: AGENTCI_MOCK_FLAKY=1 ciagent test --mock --runs 3. Details: docs/stability.md.

Audit the judge itself

An LLM judge that shares your agent's context inherits your agent's blind spots: when retrieval comes up empty, the agent answers from nothing — and the judge, reading the same nothing, passes it. judge-audit measures your judge against ground truth you already have, by re-scoring recorded baselines (the agent is never re-run):

ciagent judge-audit