artifact-driven-tdd

Artifact-Driven TDD

Philosophy

The first goal is human understanding, not code quality. You are building a system whose correctness depends on the shape of an artifact — a parsed file, an indexed dataset, a transformed table. Before any of that is worth writing, the input/output contract has to be legible to a person.

Tests here do double duty: they verify behaviour and they emit readable samples (small tables, short JSON) that make the contract concrete. A good test in this mode reads like a worked example of the contract. It runs on tiny fixtures so a human can eyeball every row.

Workflow

1. Write a human-readable test on small fixtures. 5–10 rows, hand-picked to surface the interesting cases. The assertions are secondary; the samples the test produces are primary.
2. Run it to emit readable tables/JSON. Read what comes out. This is where understanding happens — the output reveals what the contract actually is.
3. Write the formal parser/indexer from the now-understood contract. Now you know exactly what each field means, what's required, what gets rejected.
4. Run the full dataset only after the contract is agreed. Scaling up before the contract is settled just produces a large pile of wrong output.

Test categories

• Contract tests — validate invariants and red lines. Generic examples: a primary-key column must be unique and non-null; a required field may not be empty; two columns marked mutually exclusive can't both be set.
• Parser tests — validate small-file parsing on hand-picked inputs (a 10-line fixture covering comments, blanks, quoted fields — whatever the format throws at you).
• Artifact tests — validate the output schema: every required field is present, types are right, relationships between rows hold.
• Visual/notebook tests — render the behaviour as a table or chart so a human can see what the system does, not just assert it.

Anti-pattern: scaling before the contract is agreed

Do not point the pipeline at the full dataset before the contract is settled. This is horizontal slicing under another name — you're committing to a parser shape before you understand the data. The fix is the same as in /tdd: one vertical slice (one small fixture → one assertion → one understood row) at a time. Run the full dataset only once the small fixtures all pass and a human has read the samples.

Three-Piece Delivery

When a stage produces a data artifact, deliver three things together — not just the code:

1. Tests — contract, parser, artifact, and visual, per the categories above.
2. Explanatory notebook — shows the raw samples, the intermediate tables, and any policy/flag columns. A human reading it should be able to reconstruct what the stage does without reading the implementation.
3. Audit summary (a short markdown doc) — answers: what raw files were read, what artifacts were generated, which rows were excluded or quarantined (and why), and which tests exist to prevent the critical errors.

The audit summary is the human's control surface over a stage they can't hold in their head. If a stage isn't worth an audit summary, it probably isn't worth treating as artifact-driven.

Research Data Mode

When the artifact supports a research claim (a paper, a report, a thesis chapter), the three pieces above are necessary but not sufficient. A research artifact also carries the profile context that decides who may cite it and under what discipline. Record these metadata fields for the artifact (from the project's Profiles config and Active Profile Block — see /research-output-profiles):

• active_profile — the profile this stage is working under.
• producer_profile — the profile that owns the artifact (exactly one).
• intended_consumer_profiles — which profiles may read it.
• allowed_use / forbidden_use — what this artifact may and may not be used for.
• source_partition — per source (canonical / metadata-only / bridge-lookup / provenance-only).
• paper_evidence_eligible — whether it may appear as paper evidence.
• claim_support_level — none / preliminary / evidence_linked / strong / runtime_only / background_only.
• target_claim_ids — the claims (from the claim matrix) this artifact is meant to support.
• promotion_required_before_reuse — whether a stricter profile needs /artifact-promotion-audit first.

Then extend the audit summary with four research-specific outputs:

1. Claim impact — allowed | forbidden | deferred. State explicitly whether this artifact may be cited as paper evidence, and if forbidden, one line on what it must NOT be cited to support.
2. Source policy — for every source the stage reads, record allowed_use, forbidden_use, and paper_evidence_eligible. A source that is allowed for pipeline plumbing but not for paper claims must be marked as such.
3. Forbidden-use checks — tests that fail loudly if a forbidden source flows into a paper-critical artifact, or if a runtime/debug output is wired into a claim path. These are the regression guards on the claim boundary.
4. Notebook cell "What this artifact can and cannot support" — one cell, plain language, naming the strongest claim the artifact allows and the closest-looking claim it forbids. This cell is what /notebook-audit reads to issue its verdict.

The audit summary must state, in plain language: what this artifact can support, what it cannot, which Claim IDs are affected, and which profiles may consume it. These do not replace the audit summary — they extend it.

Human Understanding Gate

Before a stage that produces or transforms a data artifact is allowed to move on, the user should be able to answer all six:

1. Which raw inputs did this stage read?
2. Which artifacts did it write?
3. Which rows were excluded or quarantined, and why?
4. Which claim does this artifact support?
5. Which claim is it forbidden to support?
6. Which test catches the canonical error?

If the user cannot answer after the notebook and audit summary, do not proceed to the next stage — produce a smaller fixture or a clearer notebook first. This gate composes with /notebook-audit (which formalizes it as a verdict) and with the Active Profile Block's Human understanding gate flag (see /research-output-profiles).