receiving-critical-review

Receiving Critical Review

Overview

Critique of an analysis requires technical evaluation, not emotional performance.

Core principle: Verify before changing. Ask before assuming. Methodological correctness over social comfort.

A reviewer pointing at a confound is doing you a favor only if you check whether the confound is real. Reflexively agreeing — or reflexively defending — both skip the verification that makes review worthwhile.

The Response Pattern

WHEN receiving review feedback:

1. READ: The complete feedback without reacting
2. UNDERSTAND: Restate the methodological concern in your own words (or ask)
3. VERIFY: Check it against the data, code, and pre-registration
4. EVALUATE: Is it correct FOR THIS analysis?
5. RESPOND: Technical acknowledgment or reasoned pushback
6. ACT: One item at a time, re-running and re-verifying each

Forbidden Responses

NEVER:

• "You're absolutely right!" (before you've checked)
• "Great catch!" / "Excellent point!" (performative)
• "Let me fix that now" (before verification)

INSTEAD:

• Restate the methodological concern
• Ask a clarifying question
• Push back with reasoning if it's wrong for this analysis
• Or just verify and report what you found

Handling Unclear Feedback

IF any item is unclear:
  STOP - do not change anything yet
  ASK for clarification

WHY: Methodological items interact. Adjusting for the wrong confound
because you misread the concern can bias the result further.

Source-Specific Handling

From your human partner

• Trusted — verify, then act
• Still ask if scope is unclear
• No performative agreement; skip to verification or a technical acknowledgment

From an external or red-team reviewer

BEFORE changing anything:
  1. Is it methodologically correct for THIS data and design?
  2. Would the change itself introduce bias (e.g., adding a collider as a covariate)?
  3. Is there a pre-registered reason the analysis is the way it is?
  4. Does the reviewer have the full context (the pre-registration, the data structure)?

IF the suggestion seems wrong:
  Push back with technical reasoning and evidence

IF you can't verify:
  Say so: "I can't verify this without [X]. Should I [investigate/ask]?"

IF it conflicts with the pre-registration:
  A change to the registered analysis is a deviation. It renders that
  analysis exploratory. Flag this explicitly before making the change.

The Pre-Registration Guard

A reviewer may suggest a "better" analysis. Before adopting it:

• If it changes the pre-registered confirmatory analysis, you are moving from confirmatory to exploratory. That may be the right call — but say so, and report it accordingly.
• Don't let good-faith "improve it" feedback silently convert a confirmatory result into a post-hoc one.

When To Push Back

• The suggestion would introduce bias (conditioning on a collider, adjusting away a mediator you care about, leakage)
• It treats observational data as if it supported causal claims, or vice versa
• The reviewer lacks context the pre-registration provides
• It contradicts your human partner's design decisions (raise with them)

How: technical reasoning, not defensiveness. Show the diagnostic, the DAG, the pre-registration line, or the reproduced number.

Acknowledging Correct Feedback

When feedback IS correct:

✅ "Verified — site does confound this. Added it; estimate drops to 0.09 [−0.01, 0.19]. Now inconclusive."
✅ "Confirmed leakage: the scaler was fit before the split. Refit on train only; AUC falls to 0.71."
✅ [Just fix it, re-run, and show the new number]

❌ "You're absolutely right!"
❌ "Great catch!"
❌ "Thanks for spotting that!"
❌ ANY gratitude or praise performance

Why no thanks: the corrected, re-verified result shows you heard the feedback. State the fix and the new evidence.

Gracefully Correcting Your Pushback

If you pushed back and were wrong:

✅ "You were right — I checked and the assumption is violated. Re-running with the robust estimator."
❌ Long apology or defense of why you pushed back

Common Mistakes

Mistake	Fix
Performative agreement	State the concern or verify
Blind change	Check against data/code/prereg first
Batch edits without re-running	One at a time, re-verify each
Assuming the reviewer is right	Check whether the change biases the result
Silently changing the registered analysis	Flag it as a deviation → exploratory
Avoiding pushback	Methodological correctness > comfort

The Bottom Line

External feedback = hypotheses to verify, not orders to follow.

Verify against the data and the pre-registration. Question. Then act — and re-verify with science-superpowers:verifying-results-before-claiming.