joap-data-analysis

Data Analysis (joap-data-analysis)

JAP analyses must be model-appropriate, fully reported, and reproducible. The house toolkit is SEM/CFA, multilevel (HLM) models, mediation and moderation with proper inference, and meta-analysis. The journal expects effect sizes with confidence intervals, fit indices, bootstrap CIs for indirect effects, full disclosure of how data were handled, and a clean confirmatory vs. exploratory separation, with data and code shareable under TOP.

When to trigger

• Specifying and reporting the main and supporting analyses
• A reviewer asked for fit indices, indirect-effect CIs, robustness, or disclosure
• Reconciling preregistered analyses with exploratory follow-ups
• Preparing analysis scripts and a data/codebook for deposit

Reporting norms JAP expects

1. Measurement before structure. Report the measurement model (CFA fit: χ²/df, CFI, TLI, RMSEA, SRMR) and reliability/AVE before interpreting the structural model; report measurement invariance when comparing groups or waves.
2. Effect sizes + uncertainty. Give standardized and/or unstandardized estimates with confidence intervals for key paths — not just p-values and stars.
3. Mediation done right. Report the indirect effect with a bootstrap (or Monte Carlo) CI (not only the Baron-Kenny steps or a Sobel z); for multilevel mediation use the appropriate (e.g., 1-1-1, 2-1-1, 2-2-1) decomposition and within/between separation.
4. Multilevel correctly. Report ICC(1)/ICC(2), center predictors appropriately (group-mean vs grand-mean), model random effects, and justify the estimator; do not run OLS on nested data.
5. Full disclosure. Report how sample size was determined, all exclusions (careless responding, attrition) and reasons, all conditions, and all measures. Label confirmatory vs. exploratory.
6. Meta-analysis (if applicable). Report the model (random vs fixed), heterogeneity (Q, I²), artifact corrections, publication-bias checks, and a transparent coding protocol.
7. Reproducibility. Provide scripts and a codebook; results should regenerate from shared data in a fresh session (see joap-open-science-and-transparency).

Worked micro-example (illustrative numbers)

The servant-leadership package: a multilevel mediation in the field and a causal test in the lab.

Measurement model (field, N = 612 in 74 teams)
  CFA: χ²/df = 2.1, CFI = .96, TLI = .95, RMSEA = .045, SRMR = .04
  Reliabilities ω: leadership .91, safety .88, performance .87
  Aggregation: ICC(1) = .16, ICC(2) = .77, r_wg(j) = .85 → team-level OK
Confirmatory (preregistered) — multilevel mediation (2-2-2)
  a (leadership→safety) = .42 [.27, .57]; b (safety→performance) = .31 [.14, .48]
  Indirect = .13, 95% Monte Carlo CI [.05, .23] → mediation supported
  Direct (leadership→performance | safety) = .09 [-.06, .24], ns
Confirmatory (preregistered) — lab experiment (causal leg)
  Servant vs control on safety: d = 0.46, 95% CI [0.21, 0.71]
  H3 boundary: interaction with interdependence, ΔR² = .03, CI excludes 0
Exploratory (labeled): voice as a serial L1 mediator surfaced post hoc;
  reported as exploratory, flagged for confirmation in future work.

Why this passes JAP scrutiny: the measurement model is reported before structure; the indirect effect carries a bootstrap/Monte Carlo CI; nesting is modeled and aggregation justified; the experimental leg supplies causal warrant; and the post hoc serial path is honestly demoted to exploratory.

Analysis-stage reviewer pushback and the venue fix

Reviewer pushback	What it signals	JAP fix
"Mediation by Sobel/steps only"	outdated inference	report indirect effect + bootstrap/Monte Carlo CI
"OLS on nested data"	dependence ignored	multilevel model; report ICC, centering, random effects
"No fit indices / measurement model"	construct validity unchecked	report CFA fit + reliability before structure
"Which exclusions were preregistered?"	forking-paths concern	disclosure table: rule, count, preregistered vs post hoc, result with/without
"Is this confirmatory?"	HARKing concern	point to the preregistration; relabel post hoc analyses exploratory
"Reviewer 2 couldn't rerun your code"	reproducibility gate	ship a fresh-session run log (see open-science skill)

Calibration anchors

• Report measurement before structure: a beautiful path model on a misfitting measurement model convinces no JAP reviewer.
• Prefer estimation language with CIs to dichotomous "significant/not"; bare p-value sentences read as pre-reform.
• For mediation and cross-level effects, the interval on the carrying effect is the result — design and report so that interval is defensible.

Anti-patterns

• p-values and stars with no effect size, CI, or fit indices
• Mediation claimed via Baron-Kenny steps or Sobel z without a bootstrap/Monte Carlo CI
• Ignoring nesting (OLS on multilevel data) or aggregating without ICC/r_wg justification
• Selectively reporting conditions, measures, or exclusions (undisclosed flexibility)
• HARKing exploratory findings into confirmatory hypotheses
• Code that does not reproduce the reported numbers

Output format

【Measurement model】CFA fit + reliability + invariance reported? [Y/N]
【Main result】effect size(s) + CI(s) + meaning
【Mediation/multilevel】indirect-effect bootstrap/Monte Carlo CI; nesting modeled? [Y/N/NA]
【Disclosure】N-determination + all exclusions + all conditions + all measures? [Y/N]
【Confirmatory vs exploratory】clearly separated? [Y/N]
【Reproducible】scripts + codebook + fresh-session check? [Y/N]
【Next】joap-tables-figures

Supplementary resources

• ../../resources/external_tools.md — Mplus, lavaan, lme4/nlme, psych, metafor/metaSEM, bootstrap/Monte Carlo tools
• ../../resources/official-source-map.md — statistical and disclosure requirements