aeja-robustness

Robustness Suite (aeja-robustness)

When to trigger

• The main estimate is in hand and you need to show it is not an artifact of one specification
• A referee asks "is this robust to [alternative controls / sample / functional form / inference]?"
• The result depends on a bandwidth, a clustering choice, or a sample-selection rule that could be questioned
• You suspect specification-search concerns and want to pre-empt them

The AEJ: Applied robustness bar

AEJ: Applied referees probe whether the headline number is stable, honestly inferred, and not the product of researcher degrees of freedom. Robustness here is not a wall of regressions — it is a targeted set of checks each tied to a specific threat to the design. Map every plausible objection to the one check that answers it, and report the checks so the reader sees the estimate barely moves.

Threat to the result	The check that answers it
Omitted confounders	Oster δ / coefficient-stability bounds; added controls in steps
Specification search	a specification curve / multiverse; pre-registered primary spec
Functional form	levels vs logs, alternative outcome definitions, nonparametric version
Sample selection	drop influential units, alternative inclusion rules, balanced vs unbalanced panel
Inference too narrow	clustered SEs at the right level, wild-cluster bootstrap (few clusters), randomization inference
Design-specific fragility	DID: honest-DID bounds; RD: bandwidth/donut; IV: weak-IV-robust set
Multiple outcomes/subgroups	Romano–Wolf / List–Shaikh–Wooldridge MHT adjustment

Robustness craft

1. Lock the primary specification first. Everything else is a perturbation around it; do not present five co-equal specs and let the reader guess which is preferred.
2. One threat → one check. A robustness table should read as "here is the worry, here is the evidence it is not a problem."
3. Show stability, not just significance. The persuasive object is that the point estimate barely moves, not that it stays starred.
4. Be honest about where it weakens. A check that shifts the estimate is information; report it and bound the implication rather than hiding it.
5. Match inference to the data structure (clustering, spatial dependence, few clusters) — wrong SEs are the most common AEJ: Applied robustness failure.

Checklist

• Primary specification declared (ideally pre-registered) before perturbations
• Each robustness check mapped to a specific threat, not added for volume
• Coefficient-stability evidence (Oster δ or stepwise controls) for selection-on-unobservables
• Inference stress-tested: correct clustering level + wild-cluster/randomization inference where relevant
• Design-specific sensitivity included (honest-DID / RD bandwidth / weak-IV set)
• Multiple-hypothesis adjustment if many outcomes/subgroups
• Stability of the point estimate shown, and any check that moves it reported honestly

Anti-patterns

• A 20-column robustness table with no map from check to threat ("kitchen-sink robustness")
• Reporting only that significance survives while the point estimate wanders
• Hiding the specification that breaks the result
• Clustering at the wrong level or ignoring few-cluster bias, then claiming robustness
• Treating "added more controls and it survived" as sufficient for selection on unobservables
• Subgroup p-hacking with no MHT correction

Worked vignette (illustrative)

An IV estimate of the return to a training program is 0.11 (s.e. 0.04). The robustness suite: (i) effective F of 23 rules out weak instruments; (ii) the Anderson–Rubin 95% set is [0.04, 0.19], so inference is not weak-IV-fragile; (iii) Oster δ implies selection on unobservables would need to be 1.8× selection on observables to nullify it; (iv) wild-cluster bootstrap with 14 clusters keeps the CI away from zero; (v) dropping the largest region moves the estimate to 0.10. The point estimate barely moves — the AEJ: Applied target.

Referee pushback mapped to the robustness fix

• "This looks like specification search." → Declare the pre-registered or primary spec; show a specification curve in which the point estimate barely moves.
• "Did you cluster correctly?" → Cluster at the assignment level; with few clusters report a wild-cluster bootstrap or randomization-inference p-value.
• "Could selection on unobservables explain this?" → Report Oster δ; state how strong selection on unobservables would have to be (relative to observables) to nullify the result.

Output format

【Primary spec】declared / pre-registered? [Y/N] — estimate: ___ (s.e. ___)
【Threat → check map】selection: ___ | spec-search: ___ | form: ___ | sample: ___ | inference: ___ | design: ___
【Inference】clustering level: ___; few-cluster/randomization: ___
【Design sensitivity】honest-DID / RD bandwidth / weak-IV set: ___
【Estimate stability】range across checks: [___, ___]; checks that move it: ___
【Next step】aeja-tables-figures