jop-data-analysis

Data Analysis (jop-data-analysis)

At JOP, analysis and reproducibility are the same task: acceptance is contingent on replicability, and a JOP replication analyst re-runs your code at conditional acceptance. Write the analysis so that every number in the paper is regenerated by a script — and reported with honest uncertainty within the page budget.

When to trigger

• Setting up the estimation/analysis pipeline
• Deciding which robustness checks belong in the main text vs the Online Appendix
• A reviewer asked for additional specifications, uncertainty, or sensitivity
• Preparing numbers that must match the deposited replication package exactly

Analysis norms

• Report uncertainty, not just point estimates: CIs, SEs (clustered appropriately), and substantive effect sizes a general reader can interpret.
• Specification transparency: show the primary specification clearly; relegate the grid of alternatives to the Online Appendix, but reference it.
• Robustness that targets the threat: each check should answer a specific objection (confounding, functional form, sample, measurement), not pad the count.
• Multiple comparisons: adjust or pre-specify when testing many implications.
• Substantive interpretation: translate coefficients into quantities of interest (predicted probabilities, marginal effects) — general-interest readers want magnitudes, not just stars.

Reproducible-from-line-one (the JOP analyst will re-run this)

• One master script runs everything in order and sets the working directory once.
• Set a seed for every stochastic step (bootstrap, simulation, MCMC, jitter, sampling).
• Record software and package versions for the readme (e.g., "R 4.3.1", "Stata/MP 18.0").
• Build a codebook naming and defining every variable used in the analysis.
• Tables and figures are generated by code, never hand-edited — numbers in the text must match.

Fit the analysis to the page budget

• Lead with the result that carries the argument; do not narrate every regression.
• Move the robustness grid, balance tables, and diagnostics to the Online Appendix (≤ 25 pp).
• A Short Article (≤ 10 pp) should show one clean, decisive analysis, not a buffet.

Anti-patterns

• Numbers in the manuscript that the deposited code cannot reproduce (fails the analyst check)
• Unseeded randomness or unpinned versions ("works on my machine")
• Star-gazing with no effect sizes or uncertainty a general reader can use
• Robustness checks chosen to inflate the count rather than rebut a threat
• Cramming every specification into the main text and blowing the page budget

What a JOP analysis referee is looking for

The reviewer pool spans subfields, so an analysis only a specialist can audit reads as fragile. Map each demand to the move that satisfies it before the page count forces an ugly cut.

Referee demand	Pass move	Fail signal
Usable magnitude	Marginal effect or predicted probability with CI	Coefficient stars, no magnitude in prose
Correct uncertainty	Cluster at assignment level; randomization inference	Default SEs on clustered or experimental data
Targeted robustness	Each check named to the threat it rebuts	A grid with no mapping to objections
Multiplicity honesty	Pre-specified families; adjusted p-values	One mined "significant" interaction
Reproducibility	Master script regenerates every number	"Available on request"; drifting numbers

Worked micro-example (illustrative figures)

A hypothetical Short Article asks whether a state's adoption of automatic voter registration (AVR) raised turnout, using a staggered difference-in-differences across states. The first pass runs naive two-way fixed effects and reports a +3.1-point effect (illustrative). Because adoption is staggered, already-treated states act as forbidden controls and the estimate carries negative-weight comparisons. The JOP-credible re-analysis uses a heterogeneity-robust estimator (Callaway–Sant'Anna or Sun–Abraham), reports the group-time average as +1.8 points, 95% CI [0.4, 3.2] (illustrative), shows flat pre-trends, and clusters by state. The robustness grid goes to the Online Appendix, cited in one line of main text.

Referee pushback patterns and the JOP fix

• "Your DID uses naive TWFE on staggered adoption." Re-estimate with a heterogeneity-robust estimator, show the event-study plot, and decompose the two-way estimate so the negative-weight problem is resolved.
• "Standard errors do not reflect the design." Cluster at the assignment level — the state in the AVR example — with wild-cluster bootstrap when states are few.
• "This interaction looks fished." Show the pre-registered family and the adjusted p-value; concede a null openly.

Output format

【Primary result】estimand + magnitude + uncertainty
【Robustness】each check ↔ the threat it answers (main vs appendix)
【Reproducible】master script + seeds + pinned versions + codebook? [Y/N]
【Numbers match】text == deposited output? [Y/N]
【Page discipline】main text lean, overflow in appendix? [Y/N]
【Next】jop-tables-figures

Supplementary resources

• ../../resources/external_tools.md — estimation packages and reproducibility tooling (renv, seeds, version pinning)
• ../../resources/official-source-map.md — JOP replicability-contingent acceptance and replication-analyst check