jedpsych-data-analysis

Data Analysis (jedpsych-data-analysis)

The Journal of Educational Psychology holds analyses to the standards of a rigorous psychological research journal operating in nested educational settings. The recurring requirements are: model the nesting (students in classes in schools), report effect sizes with confidence intervals that are educationally interpretable, test the mechanism (mediation/moderation), and disclose fully under JARS. Analysis scripts and data are expected to be shareable and reproducible.

When to trigger

• Running and reporting the main and supporting analyses
• A reviewer asked for multilevel modeling, effect sizes, mechanism tests, or disclosure
• Reconciling preregistered analyses with exploratory follow-ups
• Preparing analysis scripts and a codebook for deposit

Reporting norms JEP expects

1. Respect the nesting. Use multilevel (hierarchical linear) models, SEM, or growth models that account for students nested in classrooms/schools. Cluster-robust or random-effects inference is expected; ignoring clustering deflates standard errors and is a standard JEP rejection reason.
2. Educationally meaningful effect sizes + uncertainty. Report a standardized effect (e.g., Hedges's g, a multilevel d, R²/variance explained, or a growth-rate difference) with a confidence interval, and interpret it in learning terms (e.g., months of progress, percentile shift) — not just p-values and stars.
3. Test the mechanism. JEP is theory-driven: where the hypothesis includes a learning/motivational process, fit the mediation (with appropriate multilevel mediation methods) or moderation, not only the total effect.
4. Full disclosure (JARS). Report how sample size was determined, all conditions and measures, all exclusions/attrition (with reasons and counts), missing-data handling (e.g., FIML/multiple imputation), and model specification. Confirmatory vs. exploratory must be clearly separated.
5. Appropriate inference. Justify the model; report assumptions/diagnostics and fit indices for SEM; correct for multiple comparisons across many outcomes; consider robustness to alternative specifications.

Robustness and missing data

• Show the result survives reasonable alternative specifications (covariate sets, model form, with/without exclusions). Handle attrition and missingness with principled methods (FIML, MI) and report rates by arm.

Worked micro-example (illustrative numbers)

A preregistered cluster-randomized reading-comprehension trial (48 classrooms, ~1,100 students). The confirmatory analysis is a two-level model with a pretest covariate and a preregistered mediation test.

Confirmatory (preregistered) — primary effect
  Two-level model (students within classrooms), pretest-adjusted:
  classroom-level treatment effect on transfer comprehension
  g = 0.23, 95% CI [0.06, 0.40]; ICC = 0.14; ~2.0 months of progress.
  Inference uses random classroom intercepts; SEs respect clustering.
Confirmatory (preregistered) — mechanism
  Multilevel mediation: monitoring gain mediates ~40% of the effect,
  indirect 95% CI [0.02, 0.13] (excludes 0).
Sensitivity: holds with/without the preregistered attrition exclusions
  (g 0.23 → 0.21), and under FIML for missing posttests.
Exploratory (labeled): larger effect for initially low-comprehension
  readers (ATI); reported as exploratory, flagged for future confirmation.

Why this passes JEP scrutiny: the model respects nesting; the effect carries a CI and an educational interpretation; the mechanism is tested, not asserted; the sensitivity line pre-empts the "fragile-to- exclusions" reviewer; and the ATI is honestly demoted to exploratory.

Analysis-stage reviewer pushback and the venue fix

Reviewer pushback	What it signals here	JEP fix
"You ignored clustering"	deflated SEs from nesting	refit a multilevel/random-effects model; report the ICC
"Effect size, and what does it mean for learning?"	post-reform interpretability bar	add a CI and an educational metric (months/percentile)
"Mechanism untested"	total effect without theory	fit the preregistered multilevel mediation/moderation
"Which analyses were preregistered?"	forking-paths suspicion	give the disclosure table; relabel post hoc as exploratory
"How was attrition handled?"	missing-data validity	report rates by arm; use FIML/MI; show robustness

Calibration anchors

• One well-powered, properly nested effect with a tight CI and a clear educational interpretation beats a pile of stars from a model that treated students as independent — the latter is a routine JEP reject.
• Prefer estimation language ("the intervention raised transfer comprehension by g = 0.23, ~2 months of progress, 95% CI [...]") to dichotomous "significant/not."
• Mechanism evidence is what makes the paper educational psychology rather than evaluation; budget the mediation/moderation test as a first-class result, not an afterthought.

Anti-patterns

• Treating nested students as independent (single-level OLS on clustered data)
• p-values and stars with no effect size, CI, or educational interpretation
• Reporting a total intervention effect with no test of the theorized mechanism
• Selective reporting of conditions, measures, or exclusions (undisclosed flexibility)
• Ad hoc deletion of missing data with no principled method or robustness check

Output format

【Model】multilevel / SEM / growth — nesting respected? [Y/N]
【Main result】effect size + CI + educational interpretation
【Mechanism】mediation/moderation tested as hypothesized? [Y/N/NA]
【Disclosure】N-determination + all exclusions/attrition + all measures (JARS)? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Reproducible】scripts + codebook + missing-data method? [Y/N]
【Next】jedpsych-tables-figures

Supplementary resources

• ../../resources/external_tools.md — lme4/nlme, lavaan/Mplus, mediation, metafor, effectsize, missing-data tools
• ../../resources/official-source-map.md — JARS statistical and disclosure requirements