Help us improve
Share bugs, ideas, or general feedback.
npx claudepluginhub frank-luongt/faos-skills-marketplace --plugin faos-analystHow this skill is triggered — by the user, by Claude, or both
Slash command
/faos-analyst:ab-test-setupThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
<!-- AUTO-GENERATED by export-plugins.py — DO NOT EDIT -->
Guides A/B test setup with hard gates for hypothesis validation, metrics definition, sample size calculation, assumptions checks, and execution readiness. Use before coding experiments.
Designs complete A/B test plans from hypotheses, including structured hypothesis, primary/guardrail metrics, variants, sample size, duration, success criteria, and risks.
A/B test design — produce an experiment spec with hypothesis, primary metric, MDE, sample size, run time, and decision rule. Also determines when NOT to A/B test and what to do instead. Use when asked to "design an A/B test", "should we test this", "experiment design", "how do we know if this works", "what's the sample size", or "set up an experiment".
Share bugs, ideas, or general feedback.
Ensure every A/B test is valid, rigorous, and safe before a single line of code is written.
You must have:
A valid hypothesis includes:
Before designing variants or metrics, you MUST:
Ask explicitly:
"Is this the final hypothesis we are committing to for this test?"
Do NOT proceed until confirmed.
Explicitly list assumptions about:
If assumptions are weak or violated:
Choose the simplest valid test:
Default to A/B unless there is a clear reason otherwise.
Define upfront:
Estimate:
Do NOT proceed without a realistic sample size estimate.
You may proceed to implementation only if all are true:
If any item is missing, stop and resolve it.
DO:
DO NOT:
When interpreting results:
| Result | Action |
|---|---|
| Significant positive | Consider rollout |
| Significant negative | Reject variant, document learning |
| Inconclusive | Consider more traffic or bolder change |
| Guardrail failure | Do not ship, even if primary wins |
Document:
Store records in a shared, searchable location to avoid repeated failures.
Refuse to proceed if:
Explain why and recommend next steps.
A/B testing is not about proving ideas right. It is about learning the truth with confidence.
If you feel tempted to rush, simplify, or "just try it" -- that is the signal to slow down and re-check the design.