aistats-experiments

AISTATS Experiments

Use this before submission when the empirical or simulation story is not yet locked.

Experiment audit

• Map each empirical claim to a table, figure, simulation, ablation, or robustness check.
• Include baselines that represent both ML practice and relevant statistical methods.
• Separate synthetic simulations that validate assumptions from real-data experiments that show practical relevance.
• Report uncertainty for stochastic results: repeated runs, standard errors, confidence intervals, paired tests, or bootstrap intervals when appropriate.
• Report dataset splits, preprocessing, metrics, hyperparameter search ranges, final chosen settings, selection criteria, random seeds, hardware, software versions, and runtime.
• Add ablations for the mechanism, not just cosmetic variants.
• Audit for leakage, selection bias, multiple-comparison issues, and mismatch between theoretical assumptions and empirical setup.

What experiments are for at this venue

• AISTATS experiments exist to validate theory, not to win leaderboards. One focused simulation confirming a predicted rate outweighs five extra benchmark datasets.
• The strongest design triad: a synthetic study where assumptions hold exactly, a study where they are deliberately violated, and a real-data study showing practical behavior.
• Reviewers, frequently statisticians, check whether the empirical regime — sample size, dimension, noise level — matches the asymptotic regime of the theorems. A bound proven as n grows but tested only at n = 500 invites the question of relevance.

Theory-validation design table

Theoretical claim	Matching experiment	Reject pattern avoided
Convergence rate in n	Log-log error versus n with fitted slope	"Rates asserted but never plotted"
Confidence-interval coverage	Empirical coverage across many replications	"Nominal 95 percent never verified"
Regret bound	Cumulative regret versus horizon, with the bound curve overlaid	"Bound and trajectory never compared"
Robustness to misspecification	Violation-severity sweep	"Guarantees hold under assumptions the experiments quietly break"

Vignette: a kernel conditional independence test

Suppose the paper proves finite-sample type-I error control under a boundedness assumption. The matching plan: simulate under the null at several sample sizes to verify size, sweep dependence strength for power curves, then inject heavy-tailed noise that breaks boundedness to map degradation — every panel tied to a numbered theorem or remark.

Statistical reporting floor

• Replication counts and seeds for every stochastic figure; captions must say whether bars are standard errors, confidence intervals, or quantiles.
• Report the compute actually consumed rather than vague feasibility language.

Output format

[Experiment readiness] strong / adequate / weak
[Claim -> evidence map] <claim: table/figure/simulation>
[Missing statistical evidence] <uncertainty/test/seed/baseline>
[Reproducibility gaps] <hyperparameters/compute/data/code>
[Decision-critical next run] <one experiment or simulation>