iclr-artifact-evaluation

ICLR Artifact Evaluation

Use this to prepare artifacts that make an ICLR paper reproducible and reviewable. ICLR may not run a separate artifact-badge process for every paper, so the practical bar is whether reviewers and ACs can verify the claims without identity leakage or excessive setup.

Artifact package

• Provide a minimal reproduction path for every central table or figure: command, config, seed, expected runtime, hardware, and expected output file.
• Include data provenance, preprocessing scripts, licenses, and any access restrictions.
• Separate heavy checkpoints or datasets from the core anonymized supplement when file-size limits require it; document private reviewer links clearly.
• Remove usernames, organization names, cloud buckets, Git history, API keys, and metadata that can deanonymize authors.
• Add a smoke-test script that runs in minutes and confirms environment integrity.
• Mark any unreleasable component and give a defensible reason, not a vague "proprietary" note.

ICLR-specific handling

• Submit supplementary material by the paper deadline when the current Author Guide requires it.
• During discussion, use private links or revised supplements only within current OpenReview rules.
• If a demo is useful, make it anonymous and robust to reviewer traffic, and avoid analytics that identify visitors.
• After acceptance, replace anonymous links with durable public archives or project pages.

What reviewers actually open

ICLR reviewers sample the supplement under time pressure during an open discussion everyone can read later. Optimize for the first ten minutes.

Reviewer signal	Strong artifact	Weak artifact
"Rerun the headline table?"	run_main.sh with seed, config, log	"See repo", no entry point
"Is this anonymous?"	Stripped remotes, no analytics	Demo that logs reviewer IPs
"Checkpoint matches paper?"	Hash-pinned weights + eval command	Unlabeled .pt files
"What is not covered?"	"Cannot release X, license Y"	Silent gaps read as hiding

Worked vignette

A submission proposes a self-supervised contrastive objective for graph encoders and ships a 9 GB checkpoint but no eval command. A public review asks how to reproduce Table 2 without retraining. The fix: add eval_table2.sh that loads the checkpoint, runs the frozen-encoder probe, prints the exact numbers, pin the checkpoint hash, and note in the thread that it runs in minutes on one GPU. The clean path stays public forever and reassures every later reader of the accepted paper.

Reviewer-pushback patterns

• "Anonymous link is dead." Host static files in the OpenReview supplement ZIP, not an external service that can expire mid-discussion.
• "Smoke test passes but the real run does not." Ship a longer reference log so reviewers can diff intermediate values, not only final scores.
• "Proprietary, cannot share." Replace the vague label with a synthetic-data substitute.

Output format

[Artifact status] complete / partial / risky / unavailable
[Reviewer path] <fastest route to reproduce main claim>
[Anonymity risks] <metadata, links, logs, demos>
[Release plan] anonymous review / post-acceptance public / cannot release
[Missing evidence] <commands, seeds, data, checkpoints, licenses>