smr-empirical-illustration

SMR Empirical Illustration

Use this to make the real-data section earn its place. SMR expects a methods paper to show that the method matters substantively — that using it instead of the incumbent leads to a different, better-justified conclusion about the social world. A throwaway "we also applied it to some data" section is a reviewer flag.

The "it changes the answer" standard

The illustration's job is to demonstrate consequence:

• Run the incumbent and the new method on the same data, and show where they diverge. The payoff sentence is "the standard approach would have concluded X; our method shows Y, and Y is the defensible answer because [reason tied to the method's properties]."
• Tie the divergence to the mechanism established in smr-derivation-and-properties and the regime identified in smr-simulation-studies: the data should sit in the regime where the incumbent is known to fail.
• State the substantive stake: who would have made a wrong inference, and about what, if they had used the old method? The stake makes the method consequential, not just correct.

Choosing the dataset

• Pick data that lives in the failure regime (e.g., few clusters, non-invariance across groups, informative missingness, network dependence) so the method has something to do.
• Prefer public or depositable data — SMR's availability policy expects the data and code behind the illustration to be accessible (see smr-software-and-reproducibility). If data are restricted, plan the availability statement now.
• A familiar, recognizable dataset lets readers judge the result against intuition; an exotic one forces them to trust you on both the data and the method.

What to report

Element	Purpose
Side-by-side incumbent vs. new method	Show the divergence concretely
The substantive conclusion under each	Make the stake visible
A diagnostic that the data are in the failure regime	Justify why the new method is needed here
Uncertainty for both methods	Avoid replacing one overconfident answer with another
Link to released code/data	Satisfy reproducibility expectations

Keep it an illustration, not a substantive paper

The danger runs both ways. Too thin and it is decorative; too thick and the paper becomes a substantive study that belongs in ASR/AJS (the failure flagged in smr-topic-selection). Calibrate: the illustration should be deep enough to show the method changes the answer, and no deeper. The unit of analysis is the method's behavior on real data, not a full substantive argument with its own literature.

Checklist

• Incumbent and new method are run on the same data with results side by side.
• The divergence is tied to the method's mechanism and the simulated failure regime.
• The substantive stake (who would be wrong, about what) is stated.
• A diagnostic shows the data are actually in the regime where the method is needed.
• Uncertainty is reported for both methods.
• Data are public/depositable, or a restricted-data availability plan exists.
• The section stays an illustration, not a full substantive study.

Anti-patterns

• Decorative application: the method is run, but it would not change any conclusion.
• Regime mismatch: data where the incumbent is fine, so the new method has nothing to prove.
• Substantive creep: the illustration grows into an ASR/AJS-style paper and loses methods focus.
• One-method reporting: showing only the new method's result, hiding what the incumbent would say.
• Inaccessible data with no plan: an illustration readers can never reproduce.

Output format

[Illustration status] consequential / decorative / not ready
[Dataset + regime] <data : why it sits in the failure regime>
[Divergence] <incumbent conclusion vs. new-method conclusion>
[Substantive stake] <who would have been wrong, about what>
[Reproducibility] data/code accessible? restricted-data plan?
[Next SMR skill] smr-tables-figures