Set up AI-powered visual regression testing for Storybook with intelligent diff analysis and auto-approval rules
From storybook-assistantnpx claudepluginhub flight505/storybook-assistant --plugin storybook-assistant[--chromatic] [--percy]Set up AI-powered visual regression testing for your Storybook project. This command configures intelligent visual diff analysis that understands the difference between intentional design changes and actual bugs.
Detects your environment
Asks for your preferences
Installs dependencies
@storybook/test-runnerplaywrightpixelmatch (for pixel diff)Creates configuration files
.storybook/test-runner-config.ts - Test runner setup.storybook/visual-regression.config.ts - AI analysis configscripts/visual-regression/analyze_diff.py - AI analysis engine.github/workflows/visual-regression.yml - CI integration (optional)Captures baseline screenshots
.storybook/visual-baselines/Sets up NPM scripts
npm run test:visual - Run visual regression testsnpm run test:visual:update - Update baselinesnpm run test:visual:interactive - Interactive review mode# Basic setup
/setup-visual-testing
# With Chromatic integration
/setup-visual-testing --chromatic
# With Percy integration
/setup-visual-testing --percy
Read the user's project to detect:
# Check package.json for dependencies
# Check .storybook/main.ts for configuration
# Check .github/workflows for CI setup
Use AskUserQuestion to clarify:
Integration preference:
Auto-approval rules:
CI/CD integration:
npm install --save-dev @storybook/test-runner playwright pixelmatch
npx playwright install chromium
If Python scripts needed:
pip install Pillow anthropic python-dotenv
Create .storybook/test-runner-config.ts:
import { getStoryContext } from '@storybook/test-runner';
import { analyzeVisualDiff } from './visual-regression-ai';
import { compareImages } from './visual-regression-utils';
export default {
async postRender(page, context) {
const storyContext = await getStoryContext(page, context);
const storyId = context.id;
// Capture screenshot
const screenshot = await page.screenshot({ fullPage: true });
// Compare with baseline
const baselinePath = `.storybook/visual-baselines/${storyId}.png`;
const diff = await compareImages(baselinePath, screenshot);
if (diff && diff.pixelsChanged > 0) {
// AI analysis
const analysis = await analyzeVisualDiff({
diff,
storyId,
componentName: storyContext.component,
baselinePath,
currentScreenshot: screenshot,
});
// Handle based on AI categorization
if (analysis.category === 'error') {
throw new Error(
`Visual regression detected in ${storyId}:\\n${analysis.message}`
);
} else if (analysis.category === 'warning') {
console.warn(
`Visual change detected in ${storyId}: ${analysis.message}`
);
}
// 'expected' and 'ignore' categories pass silently
}
},
};
Create .storybook/visual-regression.config.ts:
export default {
// Pixel difference threshold (0-1)
threshold: 0.01, // 1% difference triggers analysis
// Auto-approval rules
autoApprove: {
tokenChanges: true, // Auto-approve design token updates
antiAliasing: true, // Ignore anti-aliasing differences
timestamps: true, // Ignore timestamp changes
uuids: true, // Ignore UUID changes
},
// AI analysis settings
aiAnalysis: {
includeGitHistory: true,
includePRDescription: true,
includeDesignTokens: true,
lookbackDays: 7,
},
// Categorization thresholds
thresholds: {
ignore: 0.001, // < 0.1% = ignore
expected: 0.01, // < 1% = expected
warning: 0.05, // < 5% = warning
error: 0.05, // >= 5% = error
},
// Notification settings
notifications: {
onError: 'always',
onWarning: 'pr-only',
onSuccess: 'never',
},
// Baseline storage
baselinePath: '.storybook/visual-baselines/',
// Cloud storage (optional)
cloudStorage: {
provider: null, // 'chromatic', 'percy', 's3', null
config: {},
},
};
Create scripts/visual-regression/analyze_diff.py using the implementation from the visual-regression-testing skill.
Key functions:
analyze_with_context(diff, baseline, context) - Main analysiscategorize_change(change, context) - Categorization logiccheck_token_match(old_color, new_color) - Token validationget_recent_commits() - Git historyload_design_tokens() - Token systemCreate .storybook/visual-regression-utils.ts:
import { PNG } from 'pngjs';
import pixelmatch from 'pixelmatch';
import fs from 'fs/promises';
export async function compareImages(
baselinePath: string,
currentScreenshot: Buffer
): Promise<{ pixelsChanged: number; diffImage: Buffer } | null> {
try {
// Read baseline
const baselineBuffer = await fs.readFile(baselinePath);
const baseline = PNG.sync.read(baselineBuffer);
const current = PNG.sync.read(currentScreenshot);
// Ensure same dimensions
if (baseline.width !== current.width || baseline.height !== current.height) {
throw new Error('Image dimensions do not match');
}
// Create diff image
const diff = new PNG({ width: baseline.width, height: baseline.height });
// Compare
const pixelsChanged = pixelmatch(
baseline.data,
current.data,
diff.data,
baseline.width,
baseline.height,
{ threshold: 0.1 }
);
return {
pixelsChanged,
diffImage: PNG.sync.write(diff),
};
} catch (error) {
// Baseline doesn't exist - first run
return null;
}
}
export async function saveBaseline(storyId: string, screenshot: Buffer) {
const path = `.storybook/visual-baselines/${storyId}.png`;
await fs.writeFile(path, screenshot);
}
Update package.json:
{
"scripts": {
"test:visual": "test-storybook",
"test:visual:update": "test-storybook --updateSnapshots",
"test:visual:interactive": "test-storybook --watch",
"test:visual:ci": "test-storybook --ci"
}
}
# Build Storybook
npm run build-storybook
# Run test runner to capture baselines
npm run test:visual:update
Output:
Capturing baseline screenshots...
✓ Button/Primary
✓ Button/Secondary
✓ Card/Default
✓ Card/WithImage
...
✓ Captured 47 stories
Baselines saved to .storybook/visual-baselines/
If user wants CI integration, create .github/workflows/visual-regression.yml:
name: Visual Regression Testing
on:
pull_request:
branches: [main, develop]
jobs:
visual-regression:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0 # Full history for git analysis
- uses: actions/setup-node@v3
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Install Playwright browsers
run: npx playwright install chromium
- name: Build Storybook
run: npm run build-storybook
- name: Run visual regression tests
run: npm run test:visual:ci
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Upload visual regression report
if: always()
uses: actions/upload-artifact@v3
with:
name: visual-regression-report
path: .storybook/visual-regression-report/
- name: Comment PR
if: failure()
uses: actions/github-script@v6
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '⚠️ Visual regression tests failed. See [report](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details.'
})
Create .storybook/VISUAL_TESTING.md:
# Visual Regression Testing
This project uses AI-powered visual regression testing to catch UI bugs.
## Running Tests
\`\`\`bash
# Run tests
npm run test:visual
# Interactive mode (review changes)
npm run test:visual:interactive
# Update baselines
npm run test:visual:update
\`\`\`
## How It Works
1. Captures screenshots of all Storybook stories
2. Compares with baseline screenshots
3. AI analyzes differences with context (git, design tokens)
4. Categorizes as: ignore, expected, warning, or error
## Auto-Approval Rules
The AI automatically approves:
- Design token updates
- Anti-aliasing differences
- Timestamp changes
Layout shifts and unexpected color changes require manual review.
## Configuration
Edit `.storybook/visual-regression.config.ts` to adjust:
- Pixel difference threshold
- Auto-approval rules
- AI analysis settings
Present to user:
✅ Visual Regression Testing Setup Complete!
Configured:
✓ AI-powered diff analysis
✓ Baseline screenshots (47 stories)
✓ Auto-approval rules (balanced)
✓ NPM scripts
✓ CI/CD integration (GitHub Actions)
Next Steps:
1. Make a change to a component
2. Run: npm run test:visual
3. Review AI analysis results
Configuration:
- Edit: .storybook/visual-regression.config.ts
- Baselines: .storybook/visual-baselines/
- Docs: .storybook/VISUAL_TESTING.md
Useful Commands:
npm run test:visual - Run tests
npm run test:visual:interactive - Interactive review
npm run test:visual:update - Update baselines
If user selected Chromatic:
npm install --save-dev chromatic
{
"scripts": {
"chromatic": "chromatic --project-token=<token>"
}
}
// .storybook/visual-regression.config.ts
export default {
cloudStorage: {
provider: 'chromatic',
config: {
projectToken: process.env.CHROMATIC_PROJECT_TOKEN,
},
},
};
If user selected Percy:
npm install --save-dev @percy/cli @percy/storybook
{
"scripts": {
"percy": "percy storybook http://localhost:6006"
}
}
If ANTHROPIC_API_KEY not set:
❌ Error: ANTHROPIC_API_KEY environment variable not set
AI-powered analysis requires Claude API access. Please:
1. Get API key from: https://console.anthropic.com/
2. Add to .env: ANTHROPIC_API_KEY=your_key_here
3. Or run with: ANTHROPIC_API_KEY=your_key npm run test:visual
Fallback: You can use traditional pixel diff without AI analysis.
Would you like to set up without AI? [Yes/No]
If baselines don't exist:
ℹ️ No baseline screenshots found. Capturing initial baselines...
This is the first run. All screenshots will be saved as baselines.
Future runs will compare against these baselines.
Run 'npm run test:visual:update' to refresh baselines after approved changes.
If Storybook < 7:
❌ Error: Storybook 7+ required for visual regression testing
Your version: 6.5.16
Required: 7.0.0+
Please upgrade Storybook:
npm install --save-dev @storybook/react@latest
Or use legacy visual testing with Percy/Chromatic.
.storybook/visual-baselines/ to gitRelated Skills:
visual-regression-testing - Core AI analysis methodologytesting-suite - Interaction and accessibility testingci-cd-generator - CI/CD pipeline setupRelated Commands:
/setup-ci-cd - Configure full CI/CD pipeline/setup-storybook - Initial Storybook setup