oss-eval

OSS Evaluation Claude Plugin

A comprehensive Claude Code plugin implementing a 15-phase methodology for evaluating open source software frameworks with built-in safeguards against AI analysis pitfalls.

This plugin is a work in progress and is subject to change.

Attribution and Acknowledgments

The methodology used here to evaluate OSS frameworks was created by @jwforres and @n1hility, and adapted here as a Claude Code Plugin.

Overview

This plugin provides a structured, bias-resistant approach to evaluating OSS frameworks for adoption decisions. It addresses common AI analysis problems including:

• Stale Knowledge: Forces web verification of all metrics
• False Differentiation: Systematic cross-platform feature checking
• OSS/Commercial Conflation: Mandatory [OSS]/[PAID] annotations
• Complexity Overestimation: Component-level effort breakdown
• Baseline Drift: Locked comparison criteria from Phase 1
• Marketing Language: Technical translation requirements

Installation

Option 1: Plugin Marketplace (Recommended)

Install directly within Claude Code using the plugin system:

# Add the marketplace (one time)
/plugin marketplace add maxamillion/claude-oss-eval-plugin

# Install the plugin
/plugin install oss-eval

The plugin will be available immediately. Commands are namespaced as /oss-eval:command.

Option 2: Local Development

For testing or contributing to the plugin:

# Clone the repository
git clone https://github.com/maxamillion/claude-oss-eval-plugin.git

# Run Claude Code with the plugin directory
claude --plugin-dir /path/to/claude-oss-eval-plugin

Option 3: Project Scope Installation

Share the plugin with your team via git:

# Add marketplace and install with project scope
/plugin marketplace add maxamillion/claude-oss-eval-plugin
/plugin install oss-eval --scope project

This adds the plugin configuration to your project's .claude/ directory.

Plugin Management

# List installed plugins
/plugin list

# Update to latest version
/plugin update oss-eval

# Disable temporarily
/plugin disable oss-eval

# Re-enable
/plugin enable oss-eval

# Uninstall
/plugin uninstall oss-eval

Quick Start

Recommended: Single Command (Automated)

Run a complete 15-phase evaluation with a single command:

# Full automated evaluation
/oss-eval:run Python web frameworks

# With interactive checkpoints for user decisions
/oss-eval:run JavaScript state management --interactive

# Stop after Phase 6 for interim review
/oss-eval:run container orchestration --to-phase 6

# Resume an interrupted evaluation
/oss-eval:run --resume

Alternative: Manual Step-by-Step

For more control, run phases individually:

# Initialize a new evaluation
/oss-eval:start Python web frameworks

# Check current progress
/oss-eval:status

# Work through phases sequentially
/oss-eval:discover     # Phase 1
/oss-eval:analyze      # Phase 2
/oss-eval:matrix       # Phase 3
# ... continue through all 15 phases

# Generate reports
/oss-eval:report --interim  # After Phase 6
/oss-eval:report            # After Phase 15

The 15-Phase Methodology

Discovery & Analysis (Phases 1-6)

Phase	Command	Purpose
1	/oss-eval:discover	Web search discovery of candidates
2	/oss-eval:analyze	Detailed candidate analysis
3	/oss-eval:matrix	Feature comparison matrix
4	/oss-eval:licensing	License and dependency analysis
5	/oss-eval:community	Community health assessment
6	/oss-eval:risk	Quantified risk scoring

Milestone: After Phase 6, generate interim report with /oss-eval:report --interim

Technical Deep-Dive (Phases 7-9)

Phase	Command	Purpose
7	/oss-eval:architecture	Code-level architecture analysis
8	/oss-eval:requirements	Requirements alignment mapping
9	/oss-eval:gaps	Gap mitigation strategy

Integration Analysis (Phases 10-12)

Phase	Command	Purpose
10	/oss-eval:ui	UI integration assessment
11	/oss-eval:costs	Total cost of ownership
12	/oss-eval:dx	Developer experience evaluation

Validation (Phases 13-15)

Phase	Command	Purpose
13	/oss-eval:context	Product context integration
14	/oss-eval:hybrid	Hybrid strategy exploration
15	/oss-eval:validate	Adversarial review & validation

Final: Generate complete report with /oss-eval:report

Key Features

Bias Prevention Skills

Three core skills loaded automatically during evaluation:

1. bias-prevention: 12 pitfall prevention rules
2. feature-verification: Cross-platform capability checking
3. source-validation: Web source credibility verification