systemic-ingestion-normalization

Phase 4: Normalization & Systemic Ingestion

You are migrating legacy entities into the configured environment. The phase-3-environment-spec.md describes the destination; your job is to clean, normalize, and load the existing substrate into it without corrupting either side.

Preconditions

• <working-dir>/substrate-context.md exists.
• <working-dir>/phase-1-landscape-report.md exists (for the source-side asset inventory).
• <working-dir>/phase-2-taxonomy-model.md exists (for the target schema).
• <working-dir>/phase-3-environment-spec.md exists and has passed gate 3.
• The environment described by phase 3 is actually instantiated and reachable. If it is not, stop and tell the user — ingestion against a non-existent environment is not the protocol.
• Read all four files in full before starting.

Four work streams

Stream 1 — Purge Redundancies

Filter out duplicate, obsolete, or non-standard system inputs from the legacy substrate.

For each pass:

• Pass name — what's being filtered (exact duplicates / fuzzy duplicates / obsolete / non-standard / corrupted)
• Detection method — how items are identified (hash equality, fuzzy match with threshold, age beyond N days, schema-conformance check, manual review)
• Action — delete / archive / quarantine / merge / preserve-with-flag
• Counts — items examined, items matched, items actioned
• Sample — 3–5 example items that were actioned, for spot-checking

Always preserve a record of what was purged. Append-only purge logs prevent unrecoverable mistakes; never hard-delete without an archive.

Stream 2 — Enrich & Align

Apply the mandatory attributes to all legacy entities before migration.

For each enrichment pass:

• Target attribute — which phase-2 attribute is being filled
• Source — where the value comes from (existing legacy field, derivation, lookup, default, manual)
• Coverage — what fraction of legacy entities can be enriched automatically
• Manual queue — what fraction needs human input, and how it's surfaced

Critical enrichments that almost always need attention:

• Universal-schema attributes added in phase 2 that didn't exist in legacy (e.g., owner if the legacy substrate didn't track ownership)
• Status/state normalization (legacy free-text status → new enum)
• Identifier reconciliation (legacy IDs → new id space, with a mapping table preserved)
• Relationship rebuilding (legacy implicit links → explicit relationships)

For every enrichment that touches data, retain a "before/after" record so the audit can verify no value was changed silently.

Stream 3 — Execute Ingestion

Transfer items into the new environment using structured, phased batches.

Phase the ingestion explicitly:

• Phase 4a — Pilot batch: small, representative slice (1–5% of items). Run through the full validation pipeline. Inspect for errors. Adjust enrichment / validation rules if needed.
• Phase 4b — Bulk batches: remainder, in batches sized to fit memory and to allow rollback on per-batch errors. Document batch size and concurrency.
• Phase 4c — Tail / edge cases: items that failed earlier batches and were surfaced for manual handling.

For each batch, log:

• Batch ID
• Item count attempted
• Item count succeeded
• Item count rejected (with rejection reason categories)
• Wall-clock duration
• Any rollback or remediation taken

Ingestion runs MUST be idempotent — if a batch fails halfway, re-running it must not duplicate already-ingested items.

Stream 4 — Audit Integrity

Run automated and manual checks to ensure zero data corruption.

Automated checks:

• Count parity — legacy items in scope vs. new items present (allowing for purge counts)
• Identifier mapping completeness — every legacy ID maps to a new ID (or is documented as intentionally dropped)
• Schema conformance — every new entity passes phase-3 validation rules
• Relationship integrity — every relationship has both endpoints present; no orphans
• Sample fidelity — random sample of N items, deep-compared between legacy and new

Manual checks:

• Stakeholder spot-check — operational role(s) named in substrate-context.md review a small sample and confirm fidelity
• Edge-case review — any item that required special handling during enrichment is reviewed by a human

Integrity audit must produce a verdict: PASS / PASS-WITH-CAVEATS / FAIL. A FAIL verdict means ingestion is rolled back to the pre-ingestion snapshot.

Composing phase-4-ingestion-report.md

Combine the four streams into a single file at <working-dir>/phase-4-ingestion-report.md. Structure:

# Phase 4 — Ingestion Report

**Substrate:** <name>
**Date range:** YYYY-MM-DD to YYYY-MM-DD
**Preconditions:** Read substrate-context.md, phase-1, phase-2, phase-3
**Postconditions:** Ready for Phase 5 (governance-evolution-protocol)

## 1. Purge passes

[per-pass summaries with counts]

## 2. Enrichment passes

[per-pass summaries with coverage]

## 3. Ingestion batches

### 3.1 Pilot
[batch log]

### 3.2 Bulk
[per-batch logs]

### 3.3 Tail / edge cases
[per-item or per-category log]

## 4. Integrity audit

### 4.1 Automated checks
[results table]

### 4.2 Manual checks
[stakeholder sign-offs, edge-case reviews]

### 4.3 Verdict
PASS | PASS-WITH-CAVEATS | FAIL — with reasoning

## 5. Open questions for Phase 5

[explicit list of operational concerns or governance gaps surfaced during ingestion]

Gate criteria (auditor will check)

The report passes Phase 4's gate iff:

1. At least one purge pass is documented (even if zero items matched — the pass establishes that the question was asked).
2. Universal-schema attributes are enriched on 100% of ingested entities — no entity missing a required field.
3. Pilot batch was run before bulk batches — explicit, in that order, with errors from the pilot triggering enrichment/validation adjustments before bulk.
4. Count parity holds — legacy_in_scope − purged = new_present (within a documented tolerance, ideally 0).
5. Integrity audit verdict is present and is PASS or PASS-WITH-CAVEATS — a FAIL verdict means ingestion did not complete; the report should not exist in its final form yet.
6. Open questions for Phase 5 section is present.

Anti-patterns

• Don't skip the pilot batch. Bulk-loading without piloting is how silent corruption enters the new environment at scale.
• Don't enrich destructively. Every enrichment must be reversible from the audit log; if you can't reverse it, you can't safely re-run it.
• Don't silently drop items that fail validation. They go to a quarantine queue with a reason code, not /dev/null.
• Don't declare PASS before stakeholder spot-check. Automated checks verify schema conformance; humans verify semantic fidelity. Both are required.

See also

• references/deduplication-strategies.md — exact, fuzzy, and semantic duplicate detection
• references/integrity-audits.md — count parity, mapping completeness, sample fidelity patterns