joap-study-design

Study Design & Measurement (joap-study-design)

JAP holds measurement and design to an exacting standard. The recurring killers are common-method variance (CMV), weak causal warrants (cross-sectional single-source data), unmodeled nesting, and construct validity gaps. This skill hardens the design before data collection, where most of these problems can actually be solved.

When to trigger

• Planning a study, a multi-study package, or a measurement strategy
• Writing a preregistration / pre-analysis plan
• A reviewer questioned CMV, causal inference, measurement, nesting, or power
• Justifying sample size at the relevant level of analysis

Design standards

1. Construct validity first. Use validated measures; report reliability and, where the construct is new or contested, provide validity evidence (CFA, convergent/discriminant, measurement invariance across groups/time). A weak measure dooms an otherwise good design.
2. Earn the causal claim. Cross-sectional single-source correlation rarely suffices. Strengthen with temporal separation (multi-wave), multiple sources (self + supervisor + objective), experimental or quasi-experimental legs, or a field experiment.
3. Design against CMV. Build in procedural remedies (temporal/source/measurement separation, protected anonymity) and plan statistical checks; declare the strategy up front. Post hoc Harman's single-factor test alone is treated as insufficient at JAP.
4. Model the nesting. If employees are nested in teams/units/firms, justify N at each level, report ICC(1)/ICC(2) and r_wg for aggregated constructs, and use multilevel models — do not ignore dependence or aggregate away the structure without justification.
5. Justify sample size at the right level. Power for the effect that carries the claim (e.g., the cross-level interaction or indirect effect), not just the total N; for multilevel designs, the L2 sample size usually constrains power.

Common-method variance — the JAP design playbook

Remedy	Type	Note
Temporal separation (multi-wave)	procedural	predictor and outcome at different waves
Source separation (self + other/objective)	procedural	the strongest single defense
Measurement/context separation	procedural	different scales/formats for predictor vs outcome
Protected anonymity, balanced items	procedural	reduces consistency and acquiescence bias
Marker variable / CFA marker technique	statistical	plan a theoretically unrelated marker in advance
ULMC (unmeasured latent method construct)	statistical	report alongside, not instead of, procedural remedies

Sample-size justification — worked example (illustrative)

For the servant-leadership package, justify N at the level the hypotheses live, before collecting.

Multilevel field study (2-2-2 / 2-1-2 mediation):
  Constraint: 74 teams (L2) drives power for the team-level indirect effect.
  Power target: 80% for the indirect effect (Monte Carlo power for multilevel
                mediation), assuming a path ≈ .25, b path ≈ .30, ICC(1) ≈ .15.
  Result: target ≥ 70 teams, ~8 members each → ~560–620; we collect 612 in 74.
Lab experiment (causal leg):
  Between-subjects, two conditions; power for the interaction (H3 boundary),
  N ≈ 240 at 80%, alpha .05; fixed-N, no optional stopping.
Aggregation: report ICC(1), ICC(2), r_wg(j) to justify team-level aggregation
            of psychological safety; preregister exclusion rules.

Pre-data lockdown checklist

Degree of freedom	Lock before data?	Where it lives
Hypotheses + direction + level	yes	preregistration
Measures (all scales, all items)	yes	preregistration (prevents scale cherry-picking)
CMV remedies (procedural + planned statistical)	yes	design + preregistration
Aggregation rules (ICC/r_wg thresholds)	yes	analysis plan
Exclusion rules (careless responding, attrition)	yes	preregistration
Covariates / model form	yes	analysis plan
Exploratory analyses	allowed, labeled	reported separately, post hoc

Design-stage reviewer pushback and the venue fix

• "Cross-sectional, same-source — common method bias" → add temporal/source separation or an experimental leg; declare procedural remedies, not just a Harman's test.
• "You ignored nesting" → model multilevel structure; report ICC(1)/ICC(2)/r_wg; justify aggregation.
• "Measure validity unclear" → report reliability, CFA fit, and invariance; cite scale provenance.
• "Underpowered for the cross-level effect" → repower at the constraining level; report the Monte Carlo power analysis (handoff to joap-data-analysis).

Anti-patterns

• Cross-sectional single-source self-report as the sole evidentiary base
• CMV addressed only by a post hoc Harman's single-factor test
• Nested data analyzed as if independent, or aggregated without ICC/r_wg justification
• New or modified measures with no validity evidence
• Sample size justified by total N while the carrying effect lives at L2

Output format

【Construct validity】reliability + CFA/invariance evidence? [Y/N]
【Causal warrant】temporal / multi-source / experimental leg present? [Y/N]
【CMV】procedural remedies + planned statistical check declared? [Y/N]
【Nesting】levels, ICC/r_wg, multilevel model justified? [Y/N/NA]
【Sample size】powered for the carrying effect at the right level? [Y/N]
【Next】joap-data-analysis

Supplementary resources

• ../../resources/external_tools.md — Mplus/lavaan/lme4, Monte Carlo power, CMV-marker and invariance tools
• ../../resources/official-source-map.md — measurement, design, and reporting expectations