check

check — analysis & comparison

Process

1. Build a comparison matrix. For each paper/method under comparison, capture: method, dataset, metric, reported result, compute cost (if stated), and year.
2. Check genuine comparability first. Before drawing any conclusion, verify the papers are actually comparable — same dataset/benchmark, same eval protocol, similar scale. Flag apples-to-oranges comparisons explicitly rather than silently normalizing them.
3. Surface tradeoffs, not verdicts. State what each approach wins and loses on (e.g. accuracy vs. latency vs. training cost). A tradeoff table beats a single ranking.
4. Never declare an unqualified "winner." If asked "which is better," answer relative to a named axis: "better on X, worse on Y" — never a bare superlative.
5. Write output to _papers/ as a comparison note containing the matrix and tradeoff summary, with a Markdown link citation to every paper compared.

Anti-Rationalization

• "Paper B has a higher benchmark number, so it's better" — a higher number on a different eval setup (different dataset split, different hardware, different task difficulty) is not a valid comparison; check the setup first.
• "The user wants a simple answer" — a false single-axis verdict is worse than a two-line tradeoff table; give the real answer.
• "They're close enough to compare directly" — state the comparability caveat explicitly instead of silently assuming it away.

Evidence

A comparison is complete when: the matrix has method/dataset/metric/cost/year for each entry, comparability caveats are explicitly stated where setups differ, and every paper in the matrix is cited with a Markdown link.

Red Flags

• A "winner" stated without naming the comparison axis
• Comparing numbers across different datasets/eval protocols without flagging it
• Missing compute cost or scale info when it materially affects the comparison