ijcai-reproducibility

IJCAI Reproducibility

Use this before submission and again before camera-ready. Reopen the current reproducibility page; IJCAI's rubric and checklist can change.

Evidence map

• Target at least a credible reproducibility rating for every major result; aim for convincing when resources can be shared safely.
• For new algorithms, include a conceptual outline or pseudocode in the paper.
• For theory, state assumptions, formal claims, citations to tools, proof sketches, and proofs for novel claims.
• For datasets, cite existing sources, describe unavailable or proprietary datasets, and include or promise release of new datasets only when legally possible.
• For computational experiments, report final hyperparameters, search ranges, selection criteria, seeds or repeat strategy, software versions, compute infrastructure, and runtime.
• Add an optional "Reproducibility" section when it resolves likely reviewer doubts, but do not depend on supplementary material for central claims.

Rubric mapping by contribution shape

IJCAI's rating moves from irreproducible to credible to convincing. Because the PC spans symbolic, search, planning, KR, multi-agent, and learning work, the evidence that earns "convincing" differs by contribution. Map each result to the right column before drafting.

Contribution	Minimum for "credible"	What lifts it to "convincing"
New algorithm (search/planning/CSP)	Pseudocode plus complexity claim in body	Runnable code, instance generator, seeds, version pins
Theoretical result	Assumptions and proof sketch in body	Full appendix proofs, citations to formal tools
Multi-agent / game-theoretic	Protocol, agent counts, payoffs	Released simulator, opponent policies, seeds
Dataset contribution	Source, licensing, collection described	Public release or controlled-access path, datasheet
Learning experiment	Hyperparameters and splits in body	Code, environment file, compute, repeat strategy

Worked vignette: a multi-agent coordination paper

A coordination-protocol paper claims faster convergence but reports no seeds and ships no simulator. To reach "convincing": state the agent count sweep and payoff matrix in the body; appendix the convergence proof sketch; release an anonymized simulator with fixed seeds and the opponent policies; pin library versions. Without the simulator the result stays "credible" at best, which an IJCAI reviewer may flag as a significance discount.

Reviewer pushback and the venue-specific fix

• "Cannot tell if the proof holds." Put assumptions and a sketch in the body and full proofs in the Technical Appendix; do not bury the theorem statement.
• "No way to regenerate the instances." Ship a deterministic generator and seeds, not just result tables, since IJCAI search/planning reviewers re-run when skeptical.
• "Dataset is unavailable." Explain the legal barrier and give enough detail for in-principle reproduction; never imply a release you cannot deliver by camera-ready.
• "Reproducibility rests on the appendix." Pull the load-bearing protocol into the 7-page body because reviewers are not required to open the supplement.

Output format

[Result inventory] <claim -> evidence location>
[Rubric target] convincing / credible / currently weak
[Missing details] <algorithm/theory/data/compute/hyperparameters/seeds>
[Paper fixes] <must be in main PDF>
[Supplement fixes] <optional or supporting evidence>