Experiment Designer Skill

Experiment Designer Skill

Purpose

Produce rigorous experiment designs from product hypotheses, and interpret results with statistical and practical significance — so you can defend every decision to a sceptical engineering lead or data scientist.

Two-Phase Process

Phase 1: Experiment Design

Required inputs: hypothesis, primary metric, current baseline, minimum detectable effect (MDE), available sample size per day.

Output:

• Hypothesis restated as: "If we [change], we expect [metric] to [move by X%] because [reason]"
• Control and variant definitions
• Primary metric (one only)
• Secondary guardrail metrics (2-3 max)
• Required sample size (calculated from MDE and baseline)
• Estimated run time in days
• Pre-defined success criteria (before the test runs — no moving goalposts)
• Design risk flags: novelty effects, seasonal confounds, multiple testing issues, network effects, sample ratio mismatch risks

Phase 2: Results Interpretation

Required inputs: control results, variant results, p-value or raw numbers, run duration, any anomalies observed.

Output:

• Statistical significance assessment (p < 0.05 threshold)
• Practical significance: was the lift meaningful for the business, not just real?
• Confidence interval interpretation
• Confounding factors to investigate
• Recommendation: Ship / Iterate / Kill / Run follow-up test
• If "Iterate": specific hypotheses to test next

Quality Checks

• Never interpret results from an underpowered test without flagging it
• Always distinguish statistical from practical significance
• Flag if test was stopped early (peeking problem)
• Note if sample ratio mismatch occurred