newms-data-analysis

Data & Analysis (newms-data-analysis)

NM&S spans interpretive, content-analytic, and computational analysis under one interdisciplinary roof. The standard is the same across them: the analysis must be transparent, credible to a reader from another tradition, and matched to what the evidence can support. This skill is about inference and reporting, not study design (newms-research-design).

When to trigger

• Moving from collected data to claims, themes, measures, or results
• A reviewer asked for reliability, robustness, validation, or a clearer analytic trail
• You need to report uncertainty or limits honestly for a cross-method audience

Qualitative inference (interviews / ethnography)

• Analytic transparency: show the path from data to claim — coding/memoing process, how themes were built, and how many informants/instances support each theme (avoid "many participants felt…").
• Negative cases and disconfirmation: report instances that cut against the reading and how they were handled — the strongest signal of credible qualitative work.
• Quote-to-claim discipline: each claim is anchored to specific evidence, not an isolated vivid quote.

Content / discourse analysis

• Quantitative content analysis: report intercoder reliability with the right statistic (Krippendorff's alpha preferred for most designs), the unit of analysis, and how disagreements were resolved; report category distributions with uncertainty, not just counts.
• Interpretive discourse analysis: make the interpretive logic auditable — what features of the text warrant the reading, and what an alternative reading would require.

Computational analysis

• Validation first: report agreement between automated measures and human labels (precision/recall, F1, agreement) before interpreting model output as a finding.
• Robustness: sensitivity to preprocessing, model/hyperparameter choices, time window, and platform; show the result is not an artifact of one pipeline.
• Inference and uncertainty: report confidence/credible intervals; respect non-random API sampling; do not over-claim causality from observational trace data.

Inference honesty (all methods)

State plainly what the analysis establishes — description, association, interpretation, or (rarely) causation — and do not let verbs outrun the design. A cross-method NM&S panel reads candor as strength.

Robustness & reliability checklist by method

Method	Minimum credibility move	Common referee ask
Interviews / ethnography	analytic trail + negative cases	"How representative are these quotes?"
Quant content analysis	intercoder reliability (alpha) + unit defined	"What's your reliability?"
Discourse analysis	auditable interpretive warrant	"Why this reading not another?"
Computational	human-label validation + robustness sweep	"Did you validate the classifier?"

Worked micro-example (illustrative)

Computational: a classifier labels courier posts as "compliance" vs. "contestation."
Validation: 500 hand-coded posts → F1 = 0.84 reported before any substantive claim.
Robustness: result holds across two embeddings + two time windows; stated explicitly.
Inference framing: "posts shift toward compliance after a ranking change" = association, not proof of
  internalization; the qualitative strand supplies the mechanism (triangulation, per mixed design).

Referee pushback → NM&S-specific fix

• "How do I know the qualitative themes aren't cherry-picked?" → Supply the analytic trail, theme prevalence, and negative cases.
• "Your classifier is a black box." → Add human-label validation metrics and a robustness sweep.
• "You imply causation from observational traces." → Downgrade the verbs; report as association and say so.

Calibration anchors

• Validate before you interpret. Computational output is not a finding until it is checked against human labels.
• Report what cuts against you. Negative cases and robustness checks build more trust than a clean story.
• Match verbs to design. Description, association, interpretation, causation — name which one, and stop there.

Anti-patterns

• "Participants said…" with no count, trail, or negative cases
• Content analysis with no reliability statistic or undefined unit of analysis
• Computational results with no validation against human labels
• Robustness checks omitted, leaving the result as a single-pipeline artifact
• Causal language on observational, non-random trace data

Output format

【Method】qualitative / content-discourse / computational / mixed
【Inference type】description / association / interpretation / causation
【Credibility move】analytic trail / reliability stat / human-label validation
【Robustness】sensitivity checks / negative cases reported? [Y/N]
【Next】newms-tables-figures

Supplementary resources

• ../../resources/external_tools.md — reliability, content-analysis, and computational packages
• ../../resources/official-source-map.md — NM&S methodological breadth