cogpsych-data-analysis

Data Analysis & Model Fitting (cogpsych-data-analysis)

Cognitive Psychology holds analyses to a model-based standard: fit the formal model, compare it to rivals with principled criteria, demonstrate that parameters and models are recoverable, use mixed models or hierarchical Bayesian estimation where the design demands it, and report effect sizes with uncertainty for behavioral results — all regenerable from deposited code. This is the experiment-to-model-fit loop that defines the venue.

When to trigger

• Fitting the formal model and comparing it to rival accounts
• Running the behavioral analyses (mixed models, hierarchical Bayesian, contrasts)
• A reviewer asked for model comparison, recovery, robustness, or fuller disclosure
• Preparing analysis/model code and a data dictionary for deposit

Reporting norms Cognitive Psychology expects

1. Fit and compare models, don't just fit one. Report fit for your model and the rival(s) under matched flexibility; compare with AIC/BIC, cross-validation, or Bayes factors as appropriate, and say what the comparison licenses.
2. Show recovery. Demonstrate parameter recovery (can the fitting procedure recover known parameters from simulated data) and model recovery (does the comparison criterion pick the generating model) — without these, a fit edge is not interpretable.
3. Use the right hierarchical structure. Crossed random effects over subjects and items call for (generalized) linear mixed models; for cognitive models, hierarchical Bayesian estimation pools strength across participants. Justify the structure; don't aggregate away the variance.
4. Effect sizes + uncertainty for behavior. Report standardized/unstandardized effect sizes with confidence/credible intervals for key behavioral results, not just p-values and stars.
5. Confirmatory vs. exploratory. Separate pre-committed model comparisons and tests from exploratory model exploration; do not present a post hoc winning model as predicted.
6. Reproducible. Model and analysis code, with seeds and pinned versions, regenerate every reported fit, figure, and table in a fresh session (see cogpsych-open-science-and-transparency).

Robustness

• Show the conclusion survives reasonable alternative model specifications, priors (for Bayesian fits), and exclusion choices; report sensitivity, not a single fragile fit. Report convergence diagnostics (e.g., R-hat, ESS) for Bayesian models.

Worked micro-example (illustrative numbers)

A preregistered three-experiment recognition-memory program fitting UVSD vs. DPSD to confidence-ROC data.

Model comparison (preregistered) — pooled across Exps 1-3
  Fit (hierarchical Bayesian, matched flexibility):
    UVSD favored: dBIC = 14 vs. DPSD; Bayes factor ~ 30 in favor of UVSD
  Recovery (required): parameter recovery good (recovered d', sigma within
    credible intervals); model recovery ~ 92% correct at the design's N/trials
  Diagnostic signature: z-ROC slope 0.78, 95% CrI [0.72, 0.84], and linear
    (no reliable curvature) — the qualitative pattern UVSD predicts and DPSD
    forbids, consistent across all three experiments
Behavioral effect (mixed model)
  List-strength manipulation on d': b = 0.31, 95% CI [0.18, 0.44]
Exploratory (labeled)
  A small response-bias drift surfaced post hoc; reported as exploratory

Why this passes Cognitive Psychology scrutiny: the model is compared (not just fit), recovery makes the comparison interpretable, the qualitative signature corroborates the fit index, hierarchy respects subject/item variance, and the exploratory drift is honestly demoted.

Analysis-stage reviewer pushback and the venue fix

Reviewer pushback	What it signals here	Cognitive Psychology fix
"You only fit your model"	one-model storytelling	fit the rival under matched flexibility; report AIC/BIC/BF and what it licenses
"Better fit may be overfitting"	flexibility imbalance	add model recovery + cross-validation; penalize complexity
"Can you recover these parameters?"	identifiability doubt	run and report parameter + model recovery simulations
"Aggregated means hide variance"	wrong error structure	refit with crossed-random-effects mixed model / hierarchical Bayesian
"Is this the model you predicted?"	post hoc selection	pre-commit the comparison; relabel post hoc fits exploratory
"I can't rerun your fits"	reproducibility gate	ship seeded model code + a fresh-session run log

Calibration anchors

• A model that is fit, compared, and recovered is the unit of evidence here — a single fit with a good index but no rival and no recovery is not persuasive.
• Trust a crossed qualitative prediction over a marginal fit advantage; report both and lead with the signature the rival forbids.
• Respect the data's hierarchy: aggregating over subjects or items inflates false positives and can bias parameter estimates; use mixed/hierarchical models and justify the random-effects structure.
• For Bayesian fits, report priors, convergence, and sensitivity — a fit without diagnostics is not reproducible evidence.

Anti-patterns

• Fitting only your model with no rival and no comparison criterion
• Claiming a fit advantage without matched flexibility, recovery, or cross-validation
• Aggregating to cell means and ignoring crossed subject/item variance
• p-values and stars with no effect size or interval for behavioral results
• Presenting a post hoc winning model as a predicted result
• Model/analysis code that does not regenerate the reported fits

Output format

【Model comparison】rivals fit under matched flexibility + criterion (AIC/BIC/BF)? [Y/N]
【Recovery】parameter + model recovery reported? [Y/N]
【Hierarchy】mixed model / hierarchical Bayesian where apt + diagnostics? [Y/N]
【Behavioral effects】effect sizes + intervals? [Y/N]
【Confirmatory vs exploratory】separated? [Y/N]
【Reproducible】seeded code + data dictionary + fresh-session check? [Y/N]
【Next】cogpsych-tables-figures

Supplementary resources

• ../../resources/external_tools.md — modeling, model-comparison, lme4/brms/Stan, JAGS, recovery simulation
• ../../resources/official-source-map.md — statistical and modeling expectations