jop-research-design

Research Design (jop-research-design)

JOP is methodologically diverse and demanding about each tradition. The design must credibly connect the argument (jop-theory-building) to evidence — and, because acceptance is contingent on replicability, it must be one a JOP replication analyst can re-run. This skill is mode-aware: pick the section matching your work and defend it against the strongest alternative.

When to trigger

• Specifying identification, case selection, or experimental design
• A reviewer questioned causal claims, case choice, external validity, or a confound
• Preparing a pre-analysis plan
• Justifying why your design adjudicates the rival account from jop-literature-positioning

Quantitative / causal inference

• Identification first. State the estimand and the assumptions that license a causal reading (ignorability, parallel trends, exclusion, continuity). Defend them, don't assert them.
• Designs: experiments (incl. survey/conjoint), DID/event study (use modern staggered-adoption estimators, not naive TWFE), IV (first-stage strength, exclusion, weak-IV-robust inference), RDD (density/manipulation tests, bandwidth robustness), matching/weighting with balance + sensitivity.
• Inference: cluster at the level of treatment assignment; randomization inference for experiments; multiple-comparison adjustment when testing many implications.
• Sensitivity: how strong must an unobserved confounder be to overturn the result?

Experiments (lab / survey / field)

• Preregister the design and primary analyses; report power/MDE; pre-specify subgroups.
• Address attention/manipulation checks, attrition, and ethics/IRB and consent.
• For survey experiments: sampling frame, treatment realism, and generalization claims.

Formal-empirical linkage

• Make the empirical test follow from the model's comparative statics, not a loose analogy.
• Distinguish predictions unique to your model from those shared with rivals.

Qualitative / case-based

• Case selection justified by design logic (typical, deviant, most/least-likely, paired comparison) — not convenience. Say what the case is a case of.
• Process tracing with explicit tests (hoop, smoking-gun, straw-in-the-wind); state what evidence would have disconfirmed the argument.
• Source transparency: plan how archives, interviews, and fieldnotes will be documented and cited (see jop-replication-and-data-policy).

The adjudication test

For the single strongest rival explanation, write one sentence: "If the rival were true rather than my argument, the data would look like ___; instead they look like ___." If you cannot, the design does not yet identify the contribution.

Design for replicability (JOP-specific)

• Choose estimators and software you can fully script — the analyst re-runs your code.
• Fix the analysis plan so results are not a moving target between drafts.
• Keep the design within the page budget: defend it crisply in the main text, push diagnostics to the Online Appendix.

Anti-patterns

• Naive TWFE on staggered treatment; clustering at the wrong level
• "Causal" language on a design that only supports association
• Convenience case selection dressed up as theory-driven
• A design whose results cannot be regenerated by a clean script (fails the replication check)

Identification objections and the JOP-credible answer

A JOP referee asks the same question of every empirical design: would the result survive if the cleanest rival story were true? Match the objection to the design move that closes it; keep diagnostics in the Online Appendix.

Referee objection	The design answer
"Leans on selection-on-observables"	Move to a counterfactual design (DID, IV, RDD) or show E-value/Oster bounds
"Parallel trends is asserted, not shown"	Event-study pre-trends plot; heterogeneity-robust estimator
"Exclusion restriction is doubtful"	Argue exclusion substantively; report first-stage F and weak-IV CIs
"Case selection is convenience"	Justify by design logic; say what the case is a case of

Worked micro-example (illustrative)

A hypothetical paper claims a new transparency law cut local corruption, identifying off staggered adoption across municipalities. A referee objects that early adopters were already cleaning up — a selection story. The author answers structurally: an event study shows flat pre-trends, a Callaway–Sant'Anna estimate gives a −0.6 SD drop, 95% CI [−1.0, −0.2] (illustrative), and an Oster bound shows an unobserved confounder would need to be 1.7× the observed covariates to null the effect (illustrative).

Referee pushback patterns and the JOP fix

• "Your causal claim outruns the design." Either downgrade the language to association or add the design feature (counterfactual, instrument, discontinuity) that licenses the causal reading.
• "This is not reproducible." Choose estimators you can fully script; the analyst re-runs the code, so a bespoke hand-tuned procedure is a liability.

Output format

【Mode】quant-causal / experiment / formal-empirical / qualitative
【Estimand or claim】what is being identified/shown
【Key assumption(s)】and how each is defended
【Rival ruled out】the adjudication sentence
【Replicable?】fully scriptable for the JOP analyst? [Y/N]
【Next】jop-data-analysis

Supplementary resources

• ../../resources/external_tools.md — design/identification packages (R/Stata/Python) and CAQDAS for qualitative work
• ../../resources/official-source-map.md — JOP methodological diversity and replicability-contingent acceptance