smr-simulation-studies

SMR Simulation Studies

Use this to build the Monte Carlo that an SMR reviewer will trust. At a methods journal the simulation is not a formality — it is the primary evidence that the analytical properties hold in finite samples and that the method beats real competitors. A weak or self-serving simulation sinks otherwise sound papers.

Design the DGP space deliberately

Reviewers attack the data-generating process first. Specify it as a designed experiment, not a convenient example:

• Factors and levels: sample size (and, for panels/networks, the relevant dimensions), the parameter that controls the difficulty (effect size, dependence, missingness rate, sparsity), and any nuisance complications. State why each level is realistic for sociological data.
• Coverage of the assumption boundary: include cells where your own assumptions fail, so the paper shows the method's limits, not just its triumphs. SMR rewards honesty about breakdown.
• Calibration to the application: at least one DGP should be calibrated to the real dataset in smr-empirical-illustration, so the simulation speaks to a setting readers care about.
• Replications and seeds: enough Monte Carlo replications for stable estimates of the metrics, with seeds fixed and reported for reproducibility.

The competitor set (non-negotiable)

A simulation that compares the new method only to a naive baseline is the classic reject. Include:

• The current default practitioners actually use.
• The strongest existing alternative for the same problem (often from a neighboring discipline — see smr-literature-positioning).
• Where relevant, an oracle / infeasible benchmark to show the gap your method closes.

If your method loses to a competitor in some cell, report it and explain when each method is preferable — conditional recommendations are more credible than universal victory.

Metrics that match the claim

Claim type	Report	Common SMR pitfall
Point estimation	bias, RMSE, relative efficiency	reporting bias but hiding variance
Inference / testing	empirical size, power, CI coverage and width	"performs well" with no coverage number
Selection / classification	accuracy + the costs of each error	accuracy only, ignoring imbalance
Computation	runtime, scaling, convergence rate	feasibility claim with no timing

Coverage and size near the nominal level are the metrics SMR reviewers scrutinize most for inference methods — report the actual numbers, not adjectives.

Presenting the study compactly

• Summarize the full grid in a table or a small-multiples figure; do not narrate every cell.
• Lead with the cell that makes the contribution's point (where the incumbent breaks and the method holds), then show the boundary where the method itself degrades.
• Hand the exhibit design to smr-tables-figures so the grid is self-contained and readable in print.

Checklist

• The DGP is specified as a factorial design with realistic levels, each justified.
• Cells where the method's own assumptions fail are included.
• At least one DGP is calibrated to the empirical illustration's data.
• The competitor set includes the current default and the strongest alternative.
• Metrics match each claim (coverage/size for inference, bias+variance for estimation).
• Replication count and seeds are reported.
• Cells where the method loses are reported with a conditional recommendation.

Anti-patterns

• Strawman comparison: only a naive baseline, never the real competitor.
• Sunny-cell selection: showing only regimes that favor the method.
• Adjective metrics: "good size control" with no rejection rates.
• Cherry-picked n: one favorable sample size with no scaling pattern.
• Uncalibrated fantasy DGP: a design unrelated to any sociological data.
• Hidden seeds / replication count: results that cannot be reproduced.

Output format

[Simulation status] convincing / needs repair / not ready
[DGP factors] <factor : levels, with realism note>
[Competitor set] <default + strongest alternative (+ oracle)>
[Metrics] <bias/RMSE/coverage/size/power/runtime as claimed>
[Boundary cell] <where the method degrades and why that is honest>
[Next SMR skill] smr-empirical-illustration