narrative-building

Scientific Narrative Builder

Instructions

1. Inquiry: Establishing Substantive Savvy

• Identify the "Invisible": Treat surveys and experiments not just as data collection, but as a process to uncover "invisible factors", specifically perceptions, knowledge, beliefs, and reasoning that administrative data cannot capture.
• Carve the Analytical Joints: Do not settle for a broad topic. Identify the specific "analytical joint" or tension where existing theory is silent or conflicted. The introduction must articulate the specific "why" of the phenomenon before proposing the "how" of the experiment.
• The ASK Framework: Druckman (2022) identifies three pathways to research questions: Assessing (observing the world and noticing puzzles), Socializing (conversations with colleagues, students, and practitioners), and Kaput (learning from failed experiments -- "the first thing one should do when an experiment fails is to ask why it failed"). The narrative should be able to trace the question's origin to one of these pathways.
• Resist the "Methods-Driven" Temptation: Ensure the research question dictates the method, not vice versa. Explicitly defend why an experiment is the necessary tool for this specific question. "It is crucial to not jump to designs for their novelty, but only to turn to them when they offer an advantage over what could otherwise have been done" (Druckman 2022). Note that "writing your survey questions is already part of the analysis stage" (Stantcheva 2023) -- question design choices are analytical decisions, not logistical ones, and the narrative should convey this.
• Contextualization: Situate the study within a "broader account of politics" or social life. The narrative must move from the general social importance to the specific theoretical gap. For policy-relevant studies, explicitly connect the research question to the target policy environment and the populations affected.
• Multi-Level Theoretical Framing: When a study includes experiments at different levels of analysis (e.g., individual-level and institutional-level), the introduction must establish the theoretical bridge between levels. State explicitly how reasoning at the micro level (e.g., fairness judgments about individual immigrants) connects to reasoning at the macro level (e.g., legitimacy judgments about governance decisions). The bridge should be conceptual, not just methodological -- explain why the same theoretical mechanism should operate at both levels.

2. Evidence: Objective Synthesis & The Logic of Trials

• Systematic Evidence Assembly: Move beyond "selective storytelling." Favor the assembly of estimates from meta-analyses or registries. If these do not exist, explicitly document the search and inclusion criteria to avoid publication bias. Study registries (EGAP, AEA, OSF) make the existence of studies visible even when their results go unpublished -- reference this universe of studies when available (Christensen et al. 2019).
• Publication Bias Acknowledgment: When reviewing prior findings, explicitly note where the published literature may overstate effect sizes due to the "file drawer problem" -- the systematic non-publication of null results. If a registered universe of studies exists, the narrative should reference it. Note where prior effect sizes may be inflated by publication bias, and frame the current study's power analysis conservatively (Druckman 2022; Lakens 2025).
• Adjudicating Theories: Clarify if the literature review is setting up a "Fact Searching" mission (estimating a causal effect) or a "Theory Testing" mission (adjudicating between competing psychological or social mechanisms). The stronger narrative frame is "explication, not demonstration" -- the goal is to explain why effects occur, not merely to show they exist (Sniderman 2018).
• The Three Modesties: Acknowledge three forms of modesty (Sniderman 2018): (1) the modesty of any single survey treatment (no single experiment is decisive), (2) the modesty of an accessible population relative to the target population (college students may not be people -- Mutz 2011), and (3) the modesty of the current design relative to the full theoretical question. Frame the study as one link in a "chain of reasoning" -- a progression of trials rather than a one-off decisive demonstration.
• Replication-Extension Strategy: When building on prior work, frame the relationship as a "replication-extension" -- the current study replicates key features of an established design while extending it to address new questions or boundary conditions. Prior studies are not just evidence to cite but designs to build upon (Druckman 2022).
• Convenience Sample Limitations: When reviewing findings from prior convenience-sample experiments (e.g., student subject pools), explicitly acknowledge that treatment effects may be heterogeneous across the population. Effects observed in narrow samples may not replicate in broader populations, and the direction or magnitude may change (Mutz 2011, citing Rashotte and Webster).
• Conflict and Bad Faith: Narratively account for potential bad faith in the existing literature, noting where prior findings may be biased by reporting incentives.
• Responding to Critique in the Narrative: When a design has been revised in response to peer or workshop feedback, the narrative should incorporate the methodological improvement as a theoretical strength, not merely an appendix note. For example, if the redesign introduces a group-threat manipulation that was absent from the original, frame this as addressing a gap in the theoretical adjudication -- "the design enables a direct test of whether fairness reasoning withstands group-threat activation" -- rather than as a correction of a flaw.

3. Transition: The "Why" Funnel

• The Funnel Structure: Organize the narrative into a "Why" funnel. Start with the general theory (the explanation of why things happen in the world) and narrow down to the specific hypothesis (the "If-Then" statement of what will be observed in this data).
• Defining the Target Population: Do not wait for the methods section to define the population. The introduction must specify the scope of the theory. Who exactly does this explanation apply to (e.g., the general citizenry vs. a specific elite subgroup)? If using a representative sample, frame this as enabling direct estimation of the Population Average Treatment Effect (PATE) rather than requiring extrapolation from a convenience sample (Mutz 2011).
• Identifying Variation: Narratively describe the "identifying variation" you intend to create. What specific "controlled variation" is being introduced to unveil the invisible factor?
• Severity Framing: Frame the study not just in terms of whether the hypothesis is confirmed, but in terms of how severe the test is -- how capable the design is of falsifying the prediction if it is wrong. A study with high severity provides more informative evidence regardless of the outcome (Lakens 2025).
• Exploratory-Confirmatory Positioning: Honestly position the study on the exploratory-confirmatory spectrum (Lakens 2025, citing Waldron and Allen 2022). Not every study needs to test pre-registered hypotheses -- pilot studies, exploratory studies, and "tightening phase" studies that build toward confirmatory tests are legitimate and valuable. The introduction should state clearly where the study falls on this spectrum.

4. Multi-Experiment Narrative Coherence

• Experiment-Level Objectives: Each experiment in a multi-experiment study should have a clear, distinct objective that the introduction previews. Avoid describing experiments as mere replications of each other -- articulate what each contributes uniquely (e.g., Experiment 1 tests micro-level fairness; Experiment 2 tests macro-level governance legitimacy).
• Cumulative Research Program: Frame the multi-experiment study as part of a cumulative research program where each experiment is "self-contained" while building on the previous one. Resources are directed toward "subsequent studies that build on what was learned in the first" rather than toward exhaustive omnibus surveys (Mutz 2011).
• Sequential Factorial Narrative: When using a "sequential factorial" design (Sniderman 2018), where Experiment 2 "splices in" additional factors to probe mechanisms discovered in Experiment 1, the narrative should preview this logic: "Experiment 1 establishes [the basic effect]; Experiment 2 introduces [the new factor] to test whether [the mechanism] holds under [the new condition]."
• Theoretical Contributions as a List: Near the end of the introduction, enumerate the study's contributions explicitly (e.g., "This study offers three key contributions: (1)... (2)... (3)..."). Each contribution should map to a specific experiment or design feature.
• Bridging Across Experiments: Write transitional language that connects experiments. The objective of Experiment 2 should explicitly reference what Experiment 1 established and explain what Experiment 2 adds. Use language like "Building on Experiment 1's focus on..., Experiment 2 shifts to..." or "Experiment 1 isolates the micro-foundations; Experiment 2 tests whether these extend to..."
• Coherence Check: After completing all design components, verify that the narrative still matches the design. If conditions were trimmed for power reasons or the design was simplified during development, the narrative must be adjusted so it does not promise more than the design can deliver (Druckman 2022).

Quality Checks

• Substantive Foundation: Is the question grounded in a real-world social or political tension?
• Invisible Factors: Does the intro name the specific belief or perception it aims to reveal?
• Analytical Joints: Is the "joint" where the theory breaks or conflicts clearly identified?
• Counterfactual Logic: Does the narrative establish what the "world without the treatment" looks like?
• Population Scope: Is the target population explicitly named in the text?
• Objective Review: Does the literature review avoid "cherry-picking" results that only support the hypothesis? Is publication bias acknowledged as a threat to the cited evidence base?
• Three Modesties: Does the narrative acknowledge the modesty of the treatment, the sample, and the design relative to the full theoretical question?
• Multi-Level Bridge: If the study spans levels of analysis, is the conceptual bridge between levels explicitly articulated?
• Experiment Differentiation: Does each experiment have a clearly stated, distinct objective?
• Contribution Enumeration: Are the study's contributions listed explicitly and mapped to design features?
• Design Improvement as Strength: If the design was revised, is the improvement framed as a theoretical advance rather than a correction?
• Exploratory-Confirmatory Position: Does the introduction state where the study falls on the exploratory-confirmatory spectrum?
• Narrative-Design Coherence: Does the introduction promise only what the final design can deliver?