A/B Testing Skill

A/B Testing Skill

Compare two URL variants with the same persona to determine which provides a better user experience. This skill guides the comparison logic and report generation.

When to Use

• Comparing a redesign against current production
• Testing feature flag variations
• Evaluating staging vs production differences
• Comparing different UX approaches

Core Concept

Multi-Persona Testing:  Same URL    + Different Personas → Compare user types
A/B Testing:            Same Persona + Different URLs    → Compare design variants

Comparison Metrics

Track these metrics for each variant:

Metric	Better When	Comparison
Tasks Completed	Higher	X/Y vs X/Y
Critical Issues	Lower	Count vs Count
Major Issues	Lower	Count vs Count
Minor Issues	Lower	Count vs Count
Confusion Events	Lower	Count vs Count
Frustration Triggers	Lower	Count vs Count

Winner Determination

Per-Metric Winner

function getMetricWinner(metricName, valueA, valueB) {
  // For issues/confusion/frustration: lower is better
  if (['criticalIssues', 'majorIssues', 'minorIssues',
       'confusionEvents', 'frustrationTriggers'].includes(metricName)) {
    if (valueA < valueB) return 'A';
    if (valueB < valueA) return 'B';
    return 'Tie';
  }

  // For task completion: higher is better
  if (metricName === 'tasksCompleted') {
    if (valueA > valueB) return 'A';
    if (valueB > valueA) return 'B';
    return 'Tie';
  }

  return 'Tie';
}

Overall Winner

function getOverallWinner(results) {
  let aWins = 0;
  let bWins = 0;

  for (const metric of results.metrics) {
    if (metric.winner === 'A') aWins++;
    if (metric.winner === 'B') bWins++;
  }

  if (aWins > bWins) return 'A';
  if (bWins > aWins) return 'B';

  // Tiebreaker: task completion
  const taskA = results.variantA.tasksCompleted;
  const taskB = results.variantB.tasksCompleted;
  if (taskA > taskB) return 'A';
  if (taskB > taskA) return 'B';

  return 'Tie';
}

Issue Categorization

Variant-Specific Issues

Issues found only in one variant:

### Variant A Only Issues
- Navigation menu hidden on mobile
- Form validation error messages unclear

### Variant B Only Issues
- New feature tooltip blocks content
- Loading spinner too subtle

Common Issues

Issues found in both variants (baseline problems):

### Common Issues (Both Variants)
- Footer links have low contrast
- No keyboard focus indicators
- Missing alt text on hero image

Common issues should be fixed regardless of which variant wins.

Screenshot Strategy

Capture comparable screenshots at the same moments:

Moment	Variant A	Variant B
Landing Page	First impression	First impression
Navigation	Menu interaction	Menu interaction
Form Start	Beginning task	Beginning task
Confusion	If/when confused	If/when confused
Error	If error occurs	If error occurs
Task Complete	Success state	Success state

Screenshot Naming Convention

ab-test-{persona}-variant-{a|b}-{moment}-{timestamp}.png

Examples:
ab-test-genz-digital-native-variant-a-landing-143022.png
ab-test-genz-digital-native-variant-b-landing-143024.png
ab-test-genz-digital-native-variant-a-confusion-143156.png

Multi-Persona A/B Testing

When testing multiple personas on both variants:

Execution Order

Option 1: Persona-first (recommended)
For each persona:
  Test Variant A
  Test Variant B
  Record preference

Option 2: Variant-first
Test all personas on Variant A
Test all personas on Variant B
Compare results

Persona-first is recommended because it keeps each persona's experience fresh and comparable.

Consensus Calculation

function getConsensus(personaResults) {
  let aPrefs = 0;
  let bPrefs = 0;
  let ties = 0;

  for (const result of personaResults) {
    if (result.prefers === 'A') aPrefs++;
    else if (result.prefers === 'B') bPrefs++;
    else ties++;
  }

  const total = personaResults.length;

  if (aPrefs > bPrefs) {
    return `Variant A preferred by ${aPrefs}/${total} personas`;
  } else if (bPrefs > aPrefs) {
    return `Variant B preferred by ${bPrefs}/${total} personas`;
  } else {
    return `Split decision: ${aPrefs} prefer A, ${bPrefs} prefer B, ${ties} tied`;
  }
}

Narration Guidelines

Starting A/B Test

"I'm going to compare two versions of this site. Let me start with Variant A..."

Transitioning Between Variants

"Okay, I've finished testing Variant A. Now let me try Variant B with fresh eyes..."

Noting Differences

"Interesting - this is different from Variant A. The navigation is much clearer here."
"Hmm, I preferred how Variant A handled this form. This version feels cluttered."

Concluding Comparison

"After testing both versions, Variant B was definitely easier to use.
The clearer navigation and faster load times made a big difference."

Quiet Mode Output

When --quiet is active:

A/B Test: [persona-id]
Variant A: [url-a]
Variant B: [url-b]

Testing Variant A...
[Screenshot: variant-a-landing.png]
[Screenshot: variant-a-complete.png]

Testing Variant B...
[Screenshot: variant-b-landing.png]
[Screenshot: variant-b-complete.png]

# Results

| Metric | A | B | Winner |
|--------|---|---|--------|
| Tasks | 2/3 | 3/3 | B |
| Critical | 1 | 0 | B |
| Major | 2 | 1 | B |

**Winner: Variant B** (3 metric wins vs 0)

Key differences:
- B: Clearer call-to-action buttons
- B: Faster page load
- A: Form validation broke on submit

Session Recording

When recording A/B tests, create separate trace files:

recordings/
├── ab-test-genz-digital-native-variant-a-2025-01-06-143022.zip
├── ab-test-genz-digital-native-variant-b-2025-01-06-143156.zip

This allows reviewing each variant's test independently at trace.playwright.dev.

Report Templates

Single Persona A/B Report

# A/B Testing Comparison Report

## Test Configuration
- **Persona**: {{PERSONA_NAME}} ({{PERSONA_ID}})
- **Variant A (Control)**: {{URL_A}}
- **Variant B (Test)**: {{URL_B}}
- **Tasks**: {{TASKS}}
- **Date**: {{DATE}}

## Results Summary

| Metric | Variant A | Variant B | Winner |
|--------|-----------|-----------|--------|
| Tasks Completed | {{A_TASKS}} | {{B_TASKS}} | {{TASKS_WINNER}} |
| Critical Issues | {{A_CRITICAL}} | {{B_CRITICAL}} | {{CRITICAL_WINNER}} |
| Major Issues | {{A_MAJOR}} | {{B_MAJOR}} | {{MAJOR_WINNER}} |
| Minor Issues | {{A_MINOR}} | {{B_MINOR}} | {{MINOR_WINNER}} |

## Overall Winner: Variant {{WINNER}}
{{WINNER_EXPLANATION}}

## Variant A Issues
{{#A_ISSUES}}
- {{ISSUE}}
{{/A_ISSUES}}

## Variant B Issues
{{#B_ISSUES}}
- {{ISSUE}}
{{/B_ISSUES}}

## Common Issues
{{#COMMON_ISSUES}}
- {{ISSUE}}
{{/COMMON_ISSUES}}

## Recommendations
{{#RECOMMENDATIONS}}
1. {{RECOMMENDATION}}
{{/RECOMMENDATIONS}}

Multi-Persona A/B Report

# Multi-Persona A/B Comparison

## Configuration
- **Variant A**: {{URL_A}}
- **Variant B**: {{URL_B}}
- **Personas Tested**: {{PERSONA_COUNT}}

## Results Matrix

| Persona | A Tasks | B Tasks | A Issues | B Issues | Prefers |
|---------|---------|---------|----------|----------|---------|
{{#PERSONA_RESULTS}}
| {{PERSONA_ID}} | {{A_TASKS}} | {{B_TASKS}} | {{A_ISSUES}} | {{B_ISSUES}} | {{PREFERS}} |
{{/PERSONA_RESULTS}}

## Consensus: {{CONSENSUS}}

{{#PERSONA_BREAKDOWNS}}
### {{PERSONA_ID}}
- **Prefers**: {{PREFERS}}
- **Reason**: {{REASON}}
{{/PERSONA_BREAKDOWNS}}

Best Practices

1. Test Same Tasks: Ensure identical tasks are tested on both variants
2. Clear Browser State: Start each variant test with fresh state
3. Capture Comparable Screenshots: Take screenshots at same moments
4. Note Subjective Differences: Some preferences can't be quantified
5. Consider All Personas: Different users may prefer different variants
6. Fix Common Issues First: Issues in both variants need fixing regardless
7. Document Assumptions: Note any differences in variant structure