A/B Test Analysis

---

name: ab-test-analysis description: Analyze A/B test results with statistical rigor — calculate significance, check guardrails, and make ship/extend/stop decisions. Use when evaluating experiment results or interpreting test data. tags: [experimentation, ab-testing, analytics, data-driven]

A/B Test Analysis

Analyze experiment results with statistical rigor and produce a clear Ship / Investigate / Extend / Stop recommendation.

This skill complements ab-test-setup (which handles experiment design). Use this skill when you have results to analyze.

---

Purpose

Most A/B test interpretations are wrong — teams either call tests too early, ignore guardrail metrics, or ship on directional trends without statistical significance. This skill enforces disciplined analysis.

---

When to Use

• An A/B test has completed its planned duration
• You have conversion data for control and variant groups
• Stakeholders are asking "did the test win?"
• You need to decide: ship, extend, or kill

When NOT to Use

• Designing or setting up an experiment (use ab-test-setup)
• The test hasn't reached minimum sample size yet
• You're analyzing observational data (not a controlled experiment)

---

Required Data (Ask If Missing)

Field	Description
Primary metric	What the test is trying to improve (e.g., conversion rate)
Control group	Sample size (N) and conversions (C) for the control
Variant group	Sample size (N) and conversions (C) for the variant
Test duration	How long the test ran
Planned duration	How long it was designed to run
Guardrail metrics	Metrics that must not degrade (e.g., revenue, page load time)
MDE	Minimum Detectable Effect used in power calculation

---

Analysis Process

Step 1: Validate the Setup

Before analyzing results, check:

• Sample size adequate? Compare actual N to planned N from power analysis
• Duration sufficient? Must cover at least 1–2 full business cycles (e.g., weekday + weekend)
• SRM check? Sample Ratio Mismatch — control and variant should have ~equal N (within 1%). If skewed, the test is invalid.
• No novelty effects? If you can, check early vs. late behavior. New UI elements get more clicks initially.

If any check fails, the test results may be unreliable. Flag this before proceeding.

Step 2: Calculate Core Statistics

For conversion rate tests:

Control conversion rate: p_c = C_control / N_control
Variant conversion rate: p_v = C_variant / N_variant
Relative lift: (p_v - p_c) / p_c × 100%

Pooled proportion: p = (C_control + C_variant) / (N_control + N_variant)
Standard error: SE = sqrt(p × (1-p) × (1/N_control + 1/N_variant))
Z-score: Z = (p_v - p_c) / SE
P-value: two-tailed from Z

95% Confidence Interval: (p_v - p_c) ± 1.96 × SE

Step 3: Assess Significance

Criterion	Threshold	Status
Statistical significance	p-value < 0.05	Pass / Fail
Practical significance	Lift > MDE	Pass / Fail
Confidence interval	Does CI exclude 0?	Pass / Fail

Both statistical AND practical significance are required to ship.

Step 4: Check Guardrail Metrics

For each guardrail metric:

Guardrail	Control	Variant	Change	Status
[metric name]	[value]	[value]	[+/- %]	OK / Warning / Degraded

A guardrail is degraded if it shows a statistically significant negative change.

Step 5: Make the Decision

Use this decision matrix:

Primary Metric	Guardrails	Recommendation
Significant positive	All OK	Ship — roll out to 100%
Significant positive	Some degraded	Investigate — understand trade-off before deciding
Not significant, positive trend	All OK	Extend — run longer if sample size was insufficient
Not significant, flat	All OK	Stop — no effect detected, free up the experiment slot
Significant negative	Any	Don't Ship — revert and learn from the result

---

Output Format

# A/B Test Results: [Test Name]

## Summary

| Field | Value |
| --- | --- |
| Test name | [name] |
| Hypothesis | [We believed X would cause Y] |
| Primary metric | [metric name] |
| Duration | [start] — [end] ([N] days) |
| Decision | **Ship / Investigate / Extend / Stop / Don't Ship** |

---

## Results

| Group | Sample Size | Conversions | Rate |
| --- | --- | --- | --- |
| Control | [N] | [C] | [rate]% |
| Variant | [N] | [C] | [rate]% |

**Relative lift:** [+/- X.X%]
**P-value:** [value]
**95% CI:** [[lower]%, [upper]%]
**Statistically significant:** Yes / No
**Practically significant:** Yes / No (MDE was [X]%)

---

## Guardrail Metrics

| Metric | Control | Variant | Change | Status |
| --- | --- | --- | --- | --- |
| [metric] | [val] | [val] | [change] | OK / Warning |

---

## Recommendation

**Decision: [Ship / Investigate / Extend / Stop / Don't Ship]**

**Rationale:** [2–3 sentences explaining the decision]

**Next steps:**
1. [action]
2. [action]

---

## Learnings

- [What we learned from this test, regardless of outcome]
- [How this informs future experiments]

---

Common Pitfalls

Pitfall	Why It's Wrong	Correct Approach
Peeking at results daily	Inflates false positive rate	Wait for planned duration and sample size
Calling it at p=0.06	"Almost significant" isn't significant	Set the threshold before the test, stick to it
Ignoring guardrails	Winning on one metric while losing on another	Always check guardrails before shipping
Post-hoc segmentation	Finding "it worked for mobile users!" after the fact is data mining	Pre-register segments or treat as hypothesis for next test
Running too many variants	Each variant needs full sample size	Limit to 1–2 variants per test
Not learning from losses	"It didn't work" is not a learning	Document WHY it didn't work and what to try next

---

Anti-Patterns

Avoid	Why	Instead
"Directional win"	Not a statistical standard	Require p < 0.05 and lift > MDE
Shipping without guardrail check	May degrade critical metrics	Always check before shipping
Ending early because it "looks good"	Sequential testing bias	Run to planned duration
Not documenting learnings	Same failed experiments get repeated	Maintain an experiment log

---

References

• Kohavi, R., Tang, D., & Xu, Y. Trustworthy Online Controlled Experiments (2020)
• Evan Miller's A/B Test Calculator
• Sample Size Calculator