experiment-design

experiment-design

Help the user produce an experiment spec that's actually launchable. Walk them through hypothesis, variations, metrics, and sample-size sanity. This skill does not write to GrowthBook — it ends with a ready-to-launch spec that experiment-launch consumes.

All API calls go through the bundled helper: ${CLAUDE_PLUGIN_ROOT}/scripts/gb-call. It needs GB_API_KEY — set in your shell, or written to ~/.config/growthbook/.env by /growthbook:gb-setup. If unset or invalid, gb-call's error message points back at /growthbook:gb-setup.

Workflow

1. Frame the hypothesis. Falsifiable, if/then/because format:

   If we change X, then Y will improve, because Z.

   Push for specificity if the hypothesis is vague. "We think users will like it" doesn't say which metric — engagement could mean five different things. "If we move the CTA above the fold, then click-through will increase, because users decide whether to engage before they scroll" gives the prediction something concrete to land against.

2. Define variations. Default to two: control (current state) and treatment (the change). Three or more variations are valid but cost statistical power; ask the user whether they really need a third. Number variations from 0 (control) to N.

3. Pick goal metrics (ideally one, two max). List available templates and available metrics:

   gb-call GET /api/v1/api/v1/experiment-templates
   gb-call GET /api/v1/metrics
   gb-call GET /api/v1/fact-metrics
   

   Help the user choose a template if one exists and applies, otherwise help the user choose metrics based on what the hypothesis predicts will move. Note the metric type (proportion, mean, ratio, quantile) — affects sample-size math. Push back at three or more goal metrics and demote the rest to secondary or guardrail: the GrowthBook decision framework treats goal metrics as plural by design, but each additional goal dilutes power and complicates the ship/kill decision.

4. Pick guardrails (1–3). Metrics that shouldn't regress. Common examples: signup rate, page error rate, latency. Push back if the user skips guardrails; often it's good for experiments to have at least 1 or 2 guardrails. Guardrails are excluded from multiple-comparison correction by design, so don't over-stack them.

5. Estimate sample size. Need three inputs from the user:

   • Baseline rate (or mean) of the primary metric
   • Minimum detectable effect (MDE) the user cares about, in relative terms ("a 2% lift in conversion").
   • Daily traffic on the affected surface.

   Use a back-of-envelope estimate to gut-check, then point the user at GrowthBook's in-app Power Calculator for the real number. A common rule-of-thumb GrowthBook documents is ≥ 200 conversions per variation for proportion metrics; the formula n ≈ 16 × p × (1 - p) / (p × MDE)^2 per variation lands in roughly the same place for 80% power. Don't quote three significant figures from either — they're estimates. Round up and surface the inputs.

   Compute the expected experiment duration: 2 × n / daily_traffic. Flag the duration on both ends:

   • > 4 weeks — likely underpowered for practical use; consider a larger MDE, a more sensitive metric, or higher-traffic surface.
   • < 1 week — risks day-of-week and weekend effects skewing the result. Recommend at least one full weekly cycle.

6. Resolve project + datasource. If the user mentions a specific project, get its ID:

   gb-call GET /api/v1/projects
   

   List datasources for context (the launch step will pick one, but worth showing the user what's available):

   gb-call GET /api/v1/data-sources
   

7. Produce the spec. Output a structured block the user can review and feed into experiment-launch:

   ## Experiment spec — <short name>
   
   **Hypothesis:** If <change>, then <outcome>, because <mechanism>.
   
   **Variations:**
   - 0: Control — <description>
   - 1: Treatment — <description>
   
   **Primary metric:** <name> (<type>) — baseline <value>
   **Guardrails:** <metric a>, <metric b>
   **MDE:** <X%>
   **Estimated sample size:** <N> per variation
   **Estimated duration:** <D> days at <T> visitors/day on the affected surface
   **Project:** <project id>
   **Tracking key suggestion:** <kebab-case-name>
   

   Ask the user to confirm before handing off to experiment-launch.

Guardrails

• Ideally one goal metric, two max. GrowthBook's decision framework treats goal metrics as plural by design and the power calculator supports up to five, but each additional goal dilutes power and complicates the ship/kill decision. Push back at three or more; demote the rest to secondary.
• At least one guardrail. Push back if the user skips guardrails. Multiple-comparison correction does not apply to guardrails (intentionally — a guardrail signal is meant to block shipping), so don't over-stack them either; 1–3 is the sweet spot.
• Falsifiable hypothesis, if/then/because format. Push the user to make the prediction concrete enough to interpret results against. "Users will engage more" doesn't say which metric.
• Sample-size math is approximate. Use it as a gut check; route the user to the in-app Power Calculator for the real number. Round up and surface the inputs you used so the user can check.
• Watch out for activation-metric bias. Activation metrics downstream of variation differences silently bias results without tripping SRM. If the user picks one, name the risk explicitly.
• Suggest an A/A test for first-time experimenters. If the org has no stopped experiments (check via flag-discovery or experiment-brainstorm), GrowthBook recommends an A/A test first to validate the implementation before running a real one.
• Day-of-week effects matter. Push back on experiment durations under one full week. Weekend traffic and behavior differ from weekday traffic.
• Don't launch from this skill. Final spec → user confirms → hand off to experiment-launch. Resist scope creep.
• Tracking-key naming is permanent. Suggest kebab-case derived from the experiment name. The launch step will use this as trackingKey; it lands in event data and can't be cleanly changed later.

Endpoints used

• GET /api/v1/metrics and /api/v1/fact-metrics — list candidate primary + guardrail metrics
• GET /api/v1/metrics/<id> — fetch baseline value for sample-size estimation
• GET /api/v1/projects — resolve project name to ID
• GET /api/v1/data-sources — list available datasources (used by launch)

Handoffs

• experiment-launch — consumes the spec and creates the draft experiment in GrowthBook.
• Manual metric creation — if the primary metric doesn't exist yet, the user needs to create it in the GrowthBook UI at <host>/metrics (or <host>/fact-tables for fact metrics) before launching. No skill for that yet; derive <host> from GB_API_URL by swapping api. → app..
• experiment-brainstorm — if the user came in without a specific hypothesis, route back here to ground a new idea in past results.