ab-testing

Ab Testing

"Method over hacks. Evidence over assumption."

TL;DR

Designs or audits an A/B test so a team can decide whether to run, fix, stop, or interpret an experiment without confusing speed with evidence. [EXPLICIT] The skill must make the hypothesis, metric contract, assumptions, sample-size needs, duration, instrumentation, risks, and decision rule explicit. [EXPLICIT]

Procedure

Step 1: Discover

• Identify the experiment goal, decision owner, user segment, traffic source, current baseline, candidate variant, and business constraint. [EXPLICIT]
• Capture the primary metric, guardrail metrics, minimum detectable effect (MDE), desired power, significance threshold, and acceptable runtime. [EXPLICIT]
• Inspect existing analytics, event names, funnel definitions, docs, or code when available. If they are missing, mark the gap instead of inventing metrics. [EXPLICIT]

Step 2: Analyze

• Convert the idea into a falsifiable hypothesis: "If we change X for audience Y, metric Z will move by N because R." [EXPLICIT]
• Check whether an A/B test is appropriate or whether discovery, analytics cleanup, usability testing, or a feature flag rollout is safer first. [EXPLICIT]
• Estimate sample-size and duration qualitatively or quantitatively from the provided baseline, traffic, variance, MDE, power, and significance inputs. If any required input is absent, return a requirements gap and a formula-ready checklist. [EXPLICIT]
• Identify validity threats: novelty effects, peeking, seasonality, sample ratio mismatch, overlapping experiments, instrumentation drift, and segment bias. [EXPLICIT]

Step 3: Execute

• Produce an experiment brief with hypothesis, variants, metric contract, sample-size assumptions, duration recommendation, launch checklist, monitoring plan, and decision rule. [EXPLICIT]
• If asked to review an existing test, classify it as ready, blocked, risky, or invalid, and name the blocking evidence. [EXPLICIT]
• Keep implementation recommendations scoped to the experiment; route broader funnel or analytics work to related skills when needed. [EXPLICIT]

Step 4: Validate

• Verify that the primary metric has one owner and one definition. [EXPLICIT]
• Verify that every recommendation is tied to provided evidence, an explicit assumption, or an open data requirement. [EXPLICIT]
• Verify that the decision rule says what happens for win, loss, inconclusive, harmed guardrail, and instrumentation failure outcomes. [EXPLICIT]
• Do not claim statistical significance, lift, ROI, or causality unless the required data and method are available. [EXPLICIT]

Quality Criteria

• Hypothesis is falsifiable and names change, audience, metric, expected movement, and rationale.
• Primary metric, guardrail metrics, event names, and data source are defined or explicitly marked as missing.
• Sample-size, MDE, power, significance, and duration assumptions are stated; absent inputs are listed as blocking requirements.
• Launch, monitoring, stopping, and decision rules are actionable.
• Risks include at least peeking, seasonality, overlapping experiments, sample ratio mismatch, and instrumentation drift when relevant.
• Claims use evidence tags or are marked as assumptions/open questions.

Anti-Patterns

Anti-Pattern	Why It's Bad	Do This Instead
Testing without a decision rule	Produces data but no decision	Define win, loss, inconclusive, and guardrail-failure actions before launch
Optimizing many primary metrics	Inflates false positives and weakens accountability	Choose one primary metric and separate guardrails
Peeking and stopping early	Makes confidence claims unreliable	Define monitoring and stopping policy before launch
Missing instrumentation checks	Invalidates results after traffic is spent	Verify events, exposure logging, and sample ratio before analysis
Treating significance as business value	A statistically detectable lift may be too small to matter	Include MDE and practical impact threshold

Related Skills

• analytics-events
• funnel-analytics
• conversion-optimization
• data-validation
• experimentation-strategy

Usage

Example invocations:

• "/ab-testing" — Run the full ab testing workflow
• "ab testing on this project" — Apply to current context

Assumptions & Limits

• Assumes access to project artifacts (code, docs, configs) [EXPLICIT]
• Does not replace domain expert judgment for final decisions [EXPLICIT]
• If baseline conversion, traffic, variance, or MDE are missing, this skill can produce a readiness brief but not a reliable sample-size claim. [EXPLICIT]

Edge Cases

Scenario	Handling
Empty or minimal input	Request clarification before proceeding
Conflicting requirements	Flag conflicts explicitly, propose resolution
Out-of-scope request	Redirect to appropriate skill or escalate

Assets

• assets/deliverable-checklist.md provides the reusable checklist for final deliverable and guardian review.