rashomon

Know whether your skills actually improve agent behavior — not just look different.

Why rashomon?

Inspired by the Rashomon effect — the idea that the same event can produce different outcomes depending on perspective. rashomon makes those differences explicit and comparable.

• Built a skill but unsure if it actually changes agent behavior?
• Iterating on skills and prompts by gut feel instead of evidence?
• Want proof that your changes made things better, not just different?

rashomon evaluates skills and prompts through blind comparison — running tasks with and without your changes in isolated environments, then comparing real outputs without knowing which version produced which.

Who Is This For?

rashomon is designed for:

• Skill authors who want evidence-based validation
• Developers using Claude Code daily
• Teams iterating on complex prompts (coding, analysis, writing)
• Anyone who wants evidence, not vibes, when improving skills and prompts

Not ideal if:

• You want one-shot prompt rewriting without comparison

Quick Example

Skill Evaluation

/recipe-eval-skill create

Creates a skill through interactive dialog, then evaluates effectiveness:

1. Collects domain knowledge, project-specific rules, and trigger phrases
2. Generates optimized skill content (graded A/B/C)
3. Runs a test task with and without the skill in isolated environments, using blind A/B comparison

What the evaluation report looks like:

Skill Quality: Grade A
- Project-specific rules clearly encoded, no critical issues

Trigger Check: pass (discovered + invoked)

Execution Effectiveness:
- Winner: with-skill
- Assessment: structural improvement
- Key difference: 3-stage catch ordering and retry constraints
  applied correctly (attributed to skill Rules 3 and 6)

Recommendation: ship

/recipe-eval-skill api-error-handling skill's scope needs adjustment

Updates an existing skill, then evaluates old vs new version side by side.

See a real-world example: I Built a Skill Reviewer. Then I Ran It on Itself.

Prompt Evaluation

/recipe-eval-prompt Write a function to sort an array

Analyzes prompt issues, generates an improved version, runs both in isolated environments, and shows what actually changed.

- BP-002 (Vague Instructions): Sort order, language, and error handling not specified
- BP-003 (Missing Output Format): No expected output structure defined

Write a TypeScript function that sorts a number array in ascending order.
- Return empty array for empty input
- Include JSDoc comments
- Output: function code with example usage

/recipe-eval-prompt Summarize this article in 3 bullet points

Installation

Requires Claude Code (this is a Claude Code plugin)

# 1. Start Claude Code
claude

# 2. Install the marketplace
/plugin marketplace add shinpr/rashomon

# 3. Install plugin
/plugin install rashomon@rashomon

# 4. Restart session (required)
# Exit and restart Claude Code

Usage

Skill Evaluation

/recipe-eval-skill create

Create a new skill and evaluate its effectiveness.

/recipe-eval-skill my-skill-name what to change

Update an existing skill and compare old vs new.

Prompt Evaluation

/recipe-eval-prompt Your prompt here

From a file:

/recipe-eval-prompt Generate code following this skill: ./prompts/my-skill.md

For complex tasks that need more time, just mention it in natural language:

/recipe-eval-prompt Refactor the entire authentication module. This might take a while.

How It Works

Skill Evaluation (/recipe-eval-skill)
    ├── skill-creator (generates/modifies skills)
    ├── skill-reviewer (grades quality A/B/C)
    ├── eval-executor ×2 (isolated worktrees)
    └── skill-eval-reporter (blind A/B comparison)