Help us improve
Share bugs, ideas, or general feedback.
From project-docs
Documents A/B tests with variants, hypothesis, measurement plan, and conclusions. Use when the user mentions running an A/B test, comparing two variants, or wanting to measure/test something systematically.
npx claudepluginhub ttorres33/teresa-torres-plugins --plugin project-docsHow this skill is triggered — by the user, by Claude, or both
Slash command
/project-docs:ab-testThis skill is limited to the following tools:
The summary Claude sees in its skill listing — used to decide when to auto-load this skill
Create and maintain structured documentation for A/B tests, capturing the hypothesis, variants, measurement approach, timeline, results, and conclusions. This ensures tests are properly documented and learnings are preserved.
Guides planning, designing, and implementing A/B tests, split tests, multivariate experiments. Covers hypotheses, sample sizes, test types, statistical principles.
Guides A/B test planning with hypothesis frameworks, statistical principles, single-variable testing, and metrics for CRO experiments.
Plans, designs, and implements A/B tests with statistical rigor, hypothesis frameworks, and sample size calculations.
Share bugs, ideas, or general feedback.
Create and maintain structured documentation for A/B tests, capturing the hypothesis, variants, measurement approach, timeline, results, and conclusions. This ensures tests are properly documented and learnings are preserved.
Invoke this skill when any of these conditions occur:
/project-docs:ab-test commandtest-name (required): Name of the A/B test (used as document title and default filename)file-path (optional): Where to save the document. Defaults to ./{test-name}.md in current working directoryDetermine file path:
./{test-name}.md in current working directoryCheck if file exists:
Walk through each section interactively:
Ask questions one at a time, waiting for the user's response before moving to the next question.
For new tests:
For existing tests (updating):
Write the document using the format below
# A/B Test: {test-name}
## Test Date
{when the test started/starts}
## Hypothesis
{what we expect to happen and why}
## Variants
### Control: {control-name}
{control description}
### Treatment: {treatment-name}
{treatment description}
## Group Assignment
{how users are assigned to control vs treatment - randomization, segmentation, percentage split}
## Measurement
{how success will be measured, what metrics}
## Results Timeline
{when results will be evaluated}
## Results
{actual results - may be empty initially}
## Conclusions
{what we learned, decision made - may be empty initially}
# A/B Test: Homepage CTA Button Size
## Test Date
January 15, 2025
## Hypothesis
We expect the larger CTA button (treatment) to increase click-through rate by 10-15% because it will be more visually prominent and easier to tap on mobile devices, where 60% of our traffic comes from.
## Variants
### Control: Standard Button
- Size: 120x40px
- Text: "Get Started"
- Color: Blue (#0066CC)
- Current conversion rate: 3.2%
### Treatment: Large Button
- Size: 160x56px
- Text: "Get Started Free"
- Color: Blue (#0066CC) with subtle gradient
- Added micro-copy below: "No credit card required"
## Group Assignment
- 50/50 random split
- Assignment via Optimizely, cookie-based on first homepage visit
- All traffic included (no geographic or device filtering)
- Users see consistent variant across sessions
## Measurement
Primary metric: Click-through rate on homepage CTA
Secondary metrics:
- Signup completion rate
- Time to first click
- Mobile vs desktop CTR difference
Data collection: Google Analytics events + Mixpanel funnel
## Results Timeline
Review after 2 weeks or 5,000 visitors per variant.
Planned review date: January 29, 2025
Statistical significance threshold: 95%
## Results
After 2 weeks (5,200 visitors per variant):
- Control CTR: 3.2%
- Treatment CTR: 4.1%
- Lift: 28% (p < 0.01)
- Mobile lift was higher (35%) than desktop (18%)
## Conclusions
The larger button significantly outperformed the control. The "No credit card required" micro-copy likely contributed to the lift beyond just size.
Decision: Roll out the larger button to 100% of traffic.
Follow-up test: Test the micro-copy separately to isolate its effect.