commres-research-design

Research Design (commres-research-design)

CR is a quantitative journal and demanding about each design. The design must credibly connect the hypotheses (commres-theory-building) to evidence and defeat the strongest rival explanation. This skill is mode-aware — pick the section that matches your study and defend it on social-science terms.

When to trigger

• Specifying an experiment, panel survey, or content-analysis protocol
• A reviewer questioned causal claims, sampling, coding reliability, validity, or a confound
• Preparing a preregistration / pre-analysis plan (note it in the cover letter)
• Justifying why your design adjudicates the rival account from commres-literature-positioning

Experiments (lab / online / survey-embedded)

• Preregister design and primary analyses; report a-priori power / MDE; pre-specify subgroups.
• Stimulus sampling: treat messages as a sample, not a fixture — multiple exemplars per condition; model message as a random factor so the feature effect is not one text's idiosyncrasy.
• Manipulation and attention checks; treatment realism; report and model attrition.
• Pre-specify the mediation/moderation test that operationalizes H2/H3; for causal mediation, prefer manipulating the mediator or a measurement-of-mediation design with stated assumptions.
• Ethics/IRB and informed consent; debrief where deception is used.

Surveys / panels

• Sampling frame, mode, and generalization claims; weighting where appropriate.
• Validated multi-item measures; report reliability (alpha/omega) and, ideally, a CFA/measurement model; guard against common-method variance (procedural and statistical remedies).
• For process claims, prefer panel (lagged) or experimental leverage — cross-sectional mediation cannot license a temporal/causal story; say so plainly if you are cross-sectional.

Content analysis

• A documented codebook; trained coders; report intercoder reliability (Krippendorff's alpha or equivalent) on an adequate subsample, and the unit of analysis.
• Justify text sampling (timeframe, sources); establish construct validity of categories, not just reliability — reliable coding of the wrong construct is still wrong.

Computational / text-as-data (when hypothesis-testing)

• Validate automated measures against human-coded gold-standard samples; report agreement.
• Document model/version, hyperparameters, seeds; report stability; do not treat outputs as ground truth.

The adjudication test (CR-specific)

For the single strongest rival explanation, write one sentence: "If the rival were true rather than my hypothesis, the data would look like ___; instead they look like ___." If you cannot, the design does not yet identify the contribution.

Reviewer-pushback patterns and the CR-specific fix

Reviewer objection	Why it lands at CR	Design-stage fix
"Single-message confound"	one stimulus cannot separate the message feature from the text	sample multiple messages per condition; model message as a random factor
"Measurement validity unclear"	a scale or coded category may not be the construct	report CFA / construct validity, not just reliability
"Common-method variance"	same-survey predictor and outcome inflate the path	procedural separation + a statistical CMV check
"Cross-sectional process claim"	mediation on one wave cannot license a causal story	move to an experiment/panel, or hedge the claim
"Effect without mechanism"	a main effect alone does not advance theory	measure and pre-specify the mediator/moderator before collection

Worked micro-example: framing survey-experiment design (illustrative)

A study claims gain- vs. loss-framed vaccine messages change intention via perceived response-efficacy, moderated by prior knowledge. A CR-defensible design: 2 (frame) × 3 (message exemplars per frame) factorial so the frame effect is estimated across six texts — defeating the single-message confound. Validated multi-item efficacy and intention scales (report alpha + a CFA), target N sized to the registered MDE, preregister the moderated-mediation model (frame → response-efficacy → intention, moderated by knowledge) with bootstrap CIs, plus an attention check. The adjudication sentence: if "any health message moves intention" were true, the gain/loss contrast would be null while overall intention rose; instead the contrast runs through efficacy and only for low-knowledge audiences — advancing framing theory rather than re-documenting persuasion.

Anti-patterns

• Causal language on a cross-sectional survey that only supports association
• Content analysis with no reported intercoder reliability or an unstated unit of analysis
• A single-message stimulus carrying a claim about a message feature
• Scales used with no reliability/validity evidence; ignoring common-method variance
• A design that cannot distinguish your hypothesis from the leading alternative

Output format

【Mode】experiment / survey-panel / content-analysis / computational
【Estimand or claim】what is being identified/tested
【Key assumption(s)】and how each is defended (incl. reliability/validity, CMV)
【Rival ruled out】the adjudication sentence
【Mediation/moderation】design supports the causal ordering? [Y/N]
【Robustness/sensitivity】planned checks
【Next】commres-data-analysis

Supplementary resources

• ../../resources/external_tools.md — design, reliability, SEM, and text-as-data packages
• ../../resources/official-source-map.md — preregistration and APA reporting notes