Stats
Actions
Tags
From icml-skills
Strengthens ICML reproducibility evidence: code/data availability, random seeds, compute disclosure, appendix evidence, and reviewer-facing claims.
How this skill is triggered — by the user, by Claude, or both
Slash command
/icml-skills:icml-reproducibilityThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Use this when the paper's acceptance risk is tied to whether experiments, code, or theory can be
Use this when the paper's acceptance risk is tied to whether experiments, code, or theory can be trusted. ICML reviewers are asked to evaluate soundness, and ICML author instructions state that reproducibility and code availability are considered in decisions.
Accepted papers may publish original supplementary material. Do not put unreleasable data, identity leaks, or private credentials in the review package. If data cannot be public, document the access path and ethics constraints in the paper.
ICML reviewers fold reproducibility into the soundness judgment rather than scoring it on a separate axis, so the question is whether a skeptical reviewer could regenerate the headline number.
| Signal a reviewer checks | Strong evidence | Weak signal that invites doubt |
|---|---|---|
| Code in review package | Anonymized, runnable, exact commands | "Code on acceptance" promise only |
| Variance | Seeds with intervals | Single run, no spread |
| Compute | Hardware and budget table | Unstated cost, unfair comparison |
| Theory | Assumptions and proof dependencies listed | Theorem with hidden conditions |
For an adaptive-optimizer paper, reproducibility means the convergence proof's assumptions are written out, the benchmark scripts run from the anonymized supplement with a fixed seed, and the compute table lets a reviewer judge whether the speedup is real or a tuning artifact. The recurring failure is a clean theorem paired with benchmark code that silently relies on an unreleased internal dataset; document the access path or move to a public dataset before the deadline.
| Pushback | ICML-specific fix |
|---|---|
| "Cannot reproduce without the code" | Ship the anonymized runnable package now, not a post-acceptance promise |
| "Hyperparameters undocumented" | Add the search protocol and final values to the appendix |
| "Speedup may be a seed artifact" | Report multiple seeds with confidence intervals |
| "Theorem assumptions hidden" | List assumptions, proof dependencies, and edge cases explicitly |
Because accepted ICML papers can publish the original supplementary material on OpenReview, the reproducibility package is also a public commitment. Confirm before the deadline that every file is releasable, every license is stated, and no private credential or identity path remains.
[Reproducibility status] strong / adequate / weak
[Weakest claim] <claim not yet supported>
[Required fix] <code/data/seed/compute/baseline/proof>
[Supplement/public-record risk] <none or issue>
[Reviewer-facing sentence] <concise reproducibility statement>
npx claudepluginhub brycewang-stanford/awesome-journal-skills --plugin icml-skillsStrengthens reproducibility for ICLR papers: maps claims to seeds, splits, commands, and compute; writes reproducibility statements and addresses reviewer concerns about verifiability.
Audits ML experiments for ICML submission/rebuttal: baselines, ablations, variance, data leakage, compute disclosure, reproducibility, negative results.
Assists with NeurIPS reproducibility: aligns Paper Checklist with the paper, writes code/data instructions, sets seed/compute disclosure, and decides MLRC vs. main track.