scaffold-agentic-evals

Scaffold Agentic Evals

Use this skill to add a local-first eval harness.

Default Stack

Load references/scaffold.md.

• Default to promptfoo for OSS, CI-friendly behavior evals.
• Ensure [email protected] is installed on PATH for both local eval commands and optional MCP use. Prefer the consuming project's existing tool provisioning: if it has flake.nix and nixpkgs provides a compatible pkgs.promptfoo, add that package there so updates flow through the flake lockfile; otherwise use the project-local package manager sandbox.
• Store configs under evals/promptfoo/.
• Store fixtures under evals/fixtures/.
• Store generated artifacts under evals/out/.
• Generate JSON, HTML, and JUnit outputs.
• Build repo-owned static reports under site/evals/.
• Do not rely on hosted sharing as the durable record.
• Pin promptfoo once the harness is committed, and disable prompt response caching for provider-backed behavior evidence.
• For coding-agent plugins, prefer Promptfoo's native harness providers such as openai:codex-sdk and anthropic:claude-agent-sdk before writing custom wrappers.
• Separate canaries that prove plugin/skill loading from natural behavior prompts that measure whether the system chooses the right guidance unaided.
• Treat Promptfoo's MCP server as optional agent tooling for validation, focused runs, and result inspection. Treat Promptfoo's mcp provider as a separate choice for evaluating MCP servers as systems under test.

Trusted And Untrusted Runs

• Run secret-free config checks on pull requests.
• Skip provider-backed live evals on untrusted PRs.
• Run live evals on trusted scheduled, manual, or main-branch workflows when required secrets are present.
• Upload artifacts on every CI run that executes evals.
• For this marketplace, point users to eval-case-reporter when they find a behavior that should become a future fixture.