Documents A/B tests with variants, hypothesis, measurement plan, and conclusions. Use when the user mentions running an A/B test, comparing two variants, or wanting to measure/test something systematically.
From project-docsnpx claudepluginhub ttorres33/teresa-torres-plugins --plugin project-docsThis skill is limited to using the following tools:
Enables AI agents to execute x402 payments with per-task budgets, spending controls, and non-custodial wallets via MCP tools. Use when agents pay for APIs, services, or other agents.
Designs and optimizes AI agent action spaces, tool definitions, observation formats, error recovery, and context for higher task completion rates.
Compares coding agents like Claude Code and Aider on custom YAML-defined codebase tasks using git worktrees, measuring pass rate, cost, time, and consistency.
Create and maintain structured documentation for A/B tests, capturing the hypothesis, variants, measurement approach, timeline, results, and conclusions. This ensures tests are properly documented and learnings are preserved.
Invoke this skill when any of these conditions occur:
/project-docs:ab-test commandtest-name (required): Name of the A/B test (used as document title and default filename)file-path (optional): Where to save the document. Defaults to ./{test-name}.md in current working directoryDetermine file path:
./{test-name}.md in current working directoryCheck if file exists:
Walk through each section interactively:
Ask questions one at a time, waiting for the user's response before moving to the next question.
For new tests:
For existing tests (updating):
Write the document using the format below
# A/B Test: {test-name}
## Test Date
{when the test started/starts}
## Hypothesis
{what we expect to happen and why}
## Variants
### Control: {control-name}
{control description}
### Treatment: {treatment-name}
{treatment description}
## Group Assignment
{how users are assigned to control vs treatment - randomization, segmentation, percentage split}
## Measurement
{how success will be measured, what metrics}
## Results Timeline
{when results will be evaluated}
## Results
{actual results - may be empty initially}
## Conclusions
{what we learned, decision made - may be empty initially}
# A/B Test: Homepage CTA Button Size
## Test Date
January 15, 2025
## Hypothesis
We expect the larger CTA button (treatment) to increase click-through rate by 10-15% because it will be more visually prominent and easier to tap on mobile devices, where 60% of our traffic comes from.
## Variants
### Control: Standard Button
- Size: 120x40px
- Text: "Get Started"
- Color: Blue (#0066CC)
- Current conversion rate: 3.2%
### Treatment: Large Button
- Size: 160x56px
- Text: "Get Started Free"
- Color: Blue (#0066CC) with subtle gradient
- Added micro-copy below: "No credit card required"
## Group Assignment
- 50/50 random split
- Assignment via Optimizely, cookie-based on first homepage visit
- All traffic included (no geographic or device filtering)
- Users see consistent variant across sessions
## Measurement
Primary metric: Click-through rate on homepage CTA
Secondary metrics:
- Signup completion rate
- Time to first click
- Mobile vs desktop CTR difference
Data collection: Google Analytics events + Mixpanel funnel
## Results Timeline
Review after 2 weeks or 5,000 visitors per variant.
Planned review date: January 29, 2025
Statistical significance threshold: 95%
## Results
After 2 weeks (5,200 visitors per variant):
- Control CTR: 3.2%
- Treatment CTR: 4.1%
- Lift: 28% (p < 0.01)
- Mobile lift was higher (35%) than desktop (18%)
## Conclusions
The larger button significantly outperformed the control. The "No credit card required" micro-copy likely contributed to the lift beyond just size.
Decision: Roll out the larger button to 100% of traffic.
Follow-up test: Test the micro-copy separately to isolate its effect.