From claude-ads
Designs A/B tests for paid advertising: structured hypotheses, sample size calculator, test duration estimator, statistical significance tables, Meta/Google/LinkedIn setup guides.
npx claudepluginhub agricidaniel/claude-ads --plugin claude-adsThis skill uses the workspace's default tool permissions.
<!-- Created: 2026-04-13 | v1.5 -->
Guides A/B test setup for ads, landing pages, emails, products: selects variables, calculates sample sizes, sets up tracking, analyzes statistical significance. Triggers on 'A/B test', 'split test' keywords.
Designs structured ad creative tests with A/B plans, multivariate strategies, sample size calculations, and iteration cadences for optimizing performance across ad platforms.
Guides planning, design, and implementation of A/B tests, split tests, and experiments using hypothesis frameworks, test types, and sample size calculations.
Share bugs, ideas, or general feedback.
Every test must start with a structured hypothesis:
IF we [change/action]
THEN [metric] will [increase/decrease] by [estimated %]
BECAUSE [reasoning based on data or insight]
Example:
IF we replace polished product shots with UGC creator videos
THEN Meta CTR will increase by 25-40%
BECAUSE Andromeda prioritizes diverse creative formats and UGC consistently outperforms polished in 2025-2026 benchmarks
Required Sample Size (per variant):
n = (Z_alpha + Z_beta)^2 × 2 × p × (1-p) / MDE^2
Where:
- Z_alpha = 1.96 (for 95% confidence)
- Z_beta = 0.84 (for 80% power)
- p = baseline conversion rate
- MDE = minimum detectable effect (relative %)
Simplified lookup:
| Baseline CVR | 5% MDE | 10% MDE | 20% MDE | 30% MDE |
|---|---|---|---|---|
| 1% | 612,000 | 153,000 | 38,300 | 17,000 |
| 2% | 302,400 | 75,600 | 18,900 | 8,400 |
| 5% | 116,800 | 29,200 | 7,300 | 3,200 |
| 10% | 55,200 | 13,800 | 3,450 | 1,530 |
| 20% | 24,600 | 6,150 | 1,540 | 680 |
Per variant, 95% confidence, 80% power
Duration = Required Sample Size / Daily Traffic per Variant
Minimum duration: 7 days (capture weekly patterns)
Maximum recommended: 28 days (avoid seasonal drift)
Learning phase: Google 7-14 days, Meta 3-7 days, LinkedIn 7-14 days
Inputs needed:
- Daily impressions or clicks
- Number of variants (2 = A/B, 3+ = multivariate)
- Baseline conversion rate
- Minimum detectable effect desired
| Daily Clicks | 2% CVR, 20% MDE | 5% CVR, 20% MDE | 10% CVR, 20% MDE |
|---|---|---|---|
| 100 | 189 days | 73 days | 35 days |
| 500 | 38 days | 15 days | 7 days |
| 1,000 | 19 days | 7 days | 4 days* |
| 5,000 | 4 days* | 2 days* | 1 day* |
*Minimum 7 days recommended regardless of sample sufficiency
## A/B Test Plan
### Hypothesis
IF [change]
THEN [metric] will [direction] by [amount]
BECAUSE [reasoning]
### Test Design
| Parameter | Value |
|-----------|-------|
| Platform | [platform] |
| Test Type | [A/B / Multivariate] |
| Variable | [what's being changed] |
| Control | [current state] |
| Variant | [proposed change] |
| Primary Metric | [KPI] |
| Traffic Split | [50/50 / other] |
### Sample Size & Duration
| Metric | Value |
|--------|-------|
| Baseline CVR | [X%] |
| MDE | [X%] |
| Required Sample | [N per variant] |
| Daily Traffic | [N clicks/day] |
| Est. Duration | [X days] |
| Min Duration | 7 days |
### Success Criteria
- Winner declared at 95% confidence
- [Primary metric] improvement of [X%]+ sustained over [Y] days
- No negative impact on [secondary metric]
### Setup Instructions
[Platform-specific step-by-step]