hypothesis-building

Causal Hypothesis Architect

Instructions

1. The Identification Challenge

• Verify FPCI Resolution: Confirm that random assignment (or the identification strategy) solves the Fundamental Problem of Causal Inference for this design (Druckman 2022).
• Four Prerequisites for Experiments: Before proceeding, verify that the design meets the four prerequisites for causal inference from experiments: (1) random assignment to conditions, (2) exclusion restriction (the only difference between conditions is the treatment itself), (3) SUTVA (Stable Unit Treatment Value Assumption -- one subject's treatment does not affect another's outcome), and (4) noninterference between units (Druckman 2022).
• Define the Data Generating Process (DGP): Before drafting the hypothesis, describe the set of rules that governs how the data is created. What are the underlying mechanics of the world being studied?
• Map the Causal Diagram: Where appropriate, draw a DAG. Identify backdoor paths and confirm whether randomization closes them (Mutz 2011).
• Close the Backdoors: State which variables must be controlled for to isolate the treatment effect. If using an experiment, explain how random assignment closes these paths (Mutz 2011).
• SATE vs. PATE: Distinguish between the Sample Average Treatment Effect (SATE) and the Population Average Treatment Effect (PATE). A convenience-sample experiment estimates a SATE; a population-based experiment on a representative sample estimates the PATE directly by design, without requiring statistical modeling or extrapolation (Mutz 2011; Druckman 2022).

2. Hypothesis Formulation

• Popperian Falsifiability: Frame the hypothesis as a "basic statement", or a specific observation that, if found to be false, would invalidate the theory.
• The Counterfactual Logic: Every hypothesis must specify a comparison. Define the "untreated" world. If the hypothesis is that $X$ causes $Y$, what is the specific state of the world where $X$ is absent? Note that in many survey experiments there may be no "pure control" -- each condition provides information, just different information (Druckman 2022). Distinguish between active control groups (which receive different information on the same topic, controlling for the act of receiving information) and passive control groups (no information), as this choice defines the counterfactual and thus the estimand (Stantcheva 2023).
• Directional Clarity: Avoid "existence" claims (e.g., "there is an effect"). Use "ordinal" claims that specify the direction (higher/lower) and, where possible, the expected functional form.
• Beat Credible Competitors: The goal of a hypothesis test is not merely to reject the null of "no effect" but to beat credible alternative explanations. Design experiments that adjudicate between competing theories -- "the point of the experiment is explication, not demonstration" (Sniderman 2018). A hypothesis is stronger when it specifies which competing theoretical account would be undermined by the predicted result.
• Null-by-Design Thinking: Consider if the research is better served by a "Null-by-Design" experiment, where the expectation is no effect unless a specific threshold of intensity is met. Such designs require expert panel review of treatment intensity before fielding (Sniderman 2018).
• Estimand Specification: Every hypothesis must map to a specific estimand -- the statistical quantity that, if estimated, would test the hypothesis. For experimental designs, this typically means specifying: (a) the treatment contrast (what is compared to what), (b) the outcome metric (probability, scale score, etc.), and (c) the model that produces the estimate (e.g., AMCE from a conjoint, ATE from a vignette experiment). A hypothesis without a named estimand is not pre-registrable. For information/pedagogical experiments, distinguish between first-stage estimands (the belief or knowledge the treatment shifts) and second-stage estimands (the policy views influenced by those beliefs), with the causal chain explicit (Stantcheva 2023).
• Information Equivalence: In survey experiments, the exclusion restriction manifests as "information equivalence" -- the assumption that a manipulation only affects the intended construct and not background beliefs. If a treatment shifts respondents' perceptions of multiple constructs simultaneously, the estimand becomes ambiguous. Name the information equivalence assumption for each treatment contrast and discuss what would violate it (Stantcheva 2023).
• SESOI Requirement: For every hypothesis test, state the Smallest Effect Size of Interest (SESOI) -- the smallest effect that would be theoretically or practically meaningful. Justify the SESOI based on (a) theoretical predictions, (b) practical significance thresholds, or (c) benchmarks from the literature. A hypothesis without a SESOI cannot be rigorously evaluated (Lakens 2025).
• Disconfirming Evidence: Beyond falsifiability in the abstract, specify concretely what pattern of results would constitute evidence against the hypothesis. For example: "If the interaction coefficients are jointly significant and indicate that procedural effects vanish when group threat is activated, this would favor group-centric accounts over the NBM."
• Three-Level Specification: Specify each hypothesis at three levels (Lakens 2025): (1) conceptual (the theoretical claim in plain language), (2) operationalized (the specific measures and contrasts), and (3) statistical (the exact test, estimand, and decision rule). The pre-analysis plan should bridge all three levels.

3. Hypothesis Testing Logic

• Choose the Test Type: Select among NHST, interval, equivalence, and minimum-effect tests based on the theoretical claim (Lakens 2025). Not every hypothesis calls for NHST.
• Equivalence Testing for Null Predictions: When a hypothesis predicts "no meaningful effect," use the TOST (Two One-Sided Tests) procedure rather than interpreting a non-significant p-value as evidence of absence. Specify the equivalence bounds in raw effect size units and justify them. The R TOSTER package implements this procedure (Lakens 2025).
• The Four-Outcome Grid: When combining NHST with equivalence testing, state in advance which of the four outcomes (inconclusive, effect present, effect absent, trivially small) would corroborate, falsify, or leave the hypothesis inconclusive (Lakens 2025).
• Severity as Evaluation Standard: Ensure the preregistered analysis has high power and the prediction is specific enough to be wrong in multiple ways (Lakens 2025).
• Compromise Power for Fixed N: When sample size is constrained by resources or population size, use a compromise power analysis that minimizes the combined Type I + Type II error rate. An alpha > 0.05 may be defensible if it reduces total error (Lakens 2025).

4. Scope and Generalization

• Defining the Target Population: A hypothesis is not universal. Explicitly name the population for whom the theory should hold. Distinguish between the target population (who the theory applies to) and the accessible population (who can be sampled). If using a convenience sample, acknowledge that the estimand is a SATE, and specify what assumptions would be needed to generalize to the PATE.
• Mechanism vs. Effect: Distinguish between a hypothesis about a causal effect (did it happen?) and a hypothesis about a causal mechanism (why did it happen?). In audit studies and some survey experiments, the design may identify the effect but not the mechanism -- a rejection letter sent to a minority applicant shows discrimination occurred but does not reveal whether it was driven by taste-based or statistical discrimination (Druckman 2022). Similarly, distinguish priming hypotheses (salience activation without new information) from information hypotheses (belief updating through new facts) -- these have different identifying assumptions, and if a prime causes learning, the identification breaks down (Stantcheva 2023).
• Experimenter Demand Effects (EDE): Respondents may form views about the experimenter's expectations and adjust responses accordingly. EDE threatens the validity of both treatment effects and descriptive statistics. Specify EDE-mitigation strategies in the design: obfuscated follow-ups, monetary incentives for questions with correct answers, neutral framing, and multi-block survey designs that separate treatment from outcome elicitation (Stantcheva 2023).
• Measurement Hypotheses: Some hypotheses concern measurement bias rather than causal effects. For example, a list experiment tests whether respondents underreport a sensitive attitude. The estimand is the difference between the direct-report prevalence and the list-experiment-estimated prevalence. Treat these as a distinct hypothesis class (Mutz 2011; Blair et al. 2020).
• Scope Conditions vs. Hypotheses: Not all theoretical expectations should be confirmatory hypotheses. When the theory predicts that effect magnitudes will vary across contexts (e.g., countries, institutional configurations) but does not make precise directional predictions about which context produces the largest effect, treat these expectations as "scope conditions" -- interpretive frameworks examined descriptively, not through confirmatory tests. This avoids the proliferation of underpowered confirmatory hypotheses while preserving theoretical richness.

5. Hypothesis Architecture in Multi-Experiment Designs

• Sequential Numbering: When a study includes multiple experiments, number hypotheses sequentially across experiments (e.g., H1--H2 for Experiment 1, H3--H5 for Experiment 2) to maintain clarity and enable cross-referencing.
• Micro-Macro Bridging: When experiments target different levels of analysis (e.g., individual-level preferences vs. institutional-level legitimacy), hypotheses should explicitly bridge across levels. State how the macro-level prediction extends or parallels the micro-level one.
• Three-Tier Hypothesis Hierarchy: Expand beyond the binary confirmatory/exploratory distinction. Classify every hypothesis as (Lakens 2025, citing JARS):
  • Primary: Both Type I and Type II error rates are controlled and minimized. These are the study's central claims.
  • Secondary: Type I error rate is controlled, but the study may be underpowered (Type II error not controlled). These are theoretically motivated but not the main focus.
  • Exploratory: Error rates are uncontrolled. These arise during analysis and must be reported with appropriate caveats. Common candidates: individual-level moderators, three-way interactions, and cross-national comparisons where the theory underdetermines the expected pattern.
• Gate-Check -- Is the Theory Ready? Before committing to confirmatory hypotheses, ask: Is the theory sufficiently developed to generate falsifiable predictions? If predictions require arbitrary choices (e.g., picking a specific effect size threshold without justification), the study may be better framed as exploratory (Lakens 2025).
• Hypothesis-to-Model Mapping: Each confirmatory hypothesis should be explicitly linked to a specific regression model in the empirical strategy. This mapping prevents "analytical degrees of freedom" -- the researcher must commit in advance to which model tests which claim. Pre-registration in a public registry (EGAP, OSF, or AsPredicted) is the institutional commitment mechanism for this mapping.
• Counter-Hypothesis Design: Where possible, design the experiment to test two competing hypotheses against each other (e.g., fairness-based reasoning vs. group-threat reasoning) rather than a single hypothesis against the null. This provides richer theoretical information regardless of the result (Sniderman 2018).
• Sequential Factorial Logic: In multi-experiment designs, consider the "sequential factorial" approach: Experiment 1 tests the basic effect; Experiment 2 "splices in" additional factors to probe mechanisms or boundary conditions. The sequential logic means later experiments build directly on earlier findings, enabling cumulative theoretical leverage (Sniderman 2018).

Quality Checks

• Falsifiability: Can I describe a specific data result that would force the rejection of this theory?
• FPCI: Have I identified how the design solves the Fundamental Problem of Causal Inference?
• Identifying Assumptions: Have I stated what must be true (random assignment, exclusion restriction, SUTVA) for this comparison to be interpreted as causal?
• Counterfactual: Is the point of comparison (the control group) clearly defined?
• Population: Is the scope of the claim limited to a specific, named population? Is the estimand a SATE or PATE?
• Estimand: Does each hypothesis name the specific statistical quantity (AMCE, ATE, interaction coefficient) that would test it?
• SESOI: Is the smallest effect size of interest stated and justified?
• Three-Level Specification: Is each hypothesis stated at the conceptual, operationalized, and statistical levels?
• Test Type: Is the appropriate test type identified (NHST, equivalence, minimum effect, or interval)?
• Disconfirming Pattern: Is the specific data pattern that would undermine the theory described?
• Competitor Theories: Does the design test the hypothesis against a credible alternative, not just the null?
• Three-Tier Classification: Are hypotheses classified as primary, secondary, or exploratory?
• Scope Conditions: Are context-dependent expectations framed as scope conditions rather than confirmatory hypotheses?
• Model Mapping: Is each hypothesis linked to a specific regression specification?
• Registry: Is the pre-analysis plan registered in a public repository (EGAP, OSF, AsPredicted)?
• Information Equivalence: Is the information equivalence assumption stated for each survey experiment treatment contrast (Stantcheva 2023)?
• EDE Mitigation: Are experimenter demand effect mitigation strategies specified (Stantcheva 2023)?
• First/Second Stage: For information experiments, are first-stage (belief-shifting) and second-stage (policy view) estimands clearly distinguished (Stantcheva 2023)?