context-crusher

Context Crusher Skill

The most expensive tokens in agent work are the ones nobody reads: 300 identical JSON rows when the schema plus three samples would do, a log where one error hides among four hundred heartbeats, a file pasted whole for one relevant section. This skill crushes those structurally — schema + head/tail samples + numeric stats for JSON arrays, dedupe-with-counts plus guaranteed error-line survival for logs, head/tail windowing for text — with a deterministic stdlib script, no model call, no summarization risk. The information that defines meaning survives; the repetition that defines cost doesn't.

What This Skill Produces

• The crushed artifact — the compressed version, with its token math in the header (~6,000 → ~130 is typical for uniform JSON)
• The crush decision — what to crush, what to keep raw, and what to link instead of load
• The escape hatch — every crush names how to fetch the original when a detail turns out to matter
• The pipeline habit — where in the agent's workflow the crush step belongs (between tool and context, always)

Required Inputs

Ask for these if not provided:

• The payload — the JSON/log/text (or its path), and roughly how it will be used ("I need the error" vs. "I need every row" are opposite answers)
• The repetition question — is this data uniform (crushable to schema+stats) or is each row genuinely distinct (crushing loses signal — keep or filter instead)?
• The journey stage — one-shot analysis (crush hard) vs. data the conversation will keep querying (crush to an index, keep the original fetchable)

Programmatic Helper

python3 scripts/context_crush.py --mode json --file response.json
python3 scripts/context_crush.py --mode log --file build.log --keep 40
cat data.json | python3 scripts/context_crush.py --mode json

Deterministic, stdlib-only, no API. JSON arrays → {count, schema, head samples, tail, numeric min/max/mean} · logs → consecutive-duplicate collapse + first-occurrence dedupe + an always-preserved error/warning section · text → whitespace normalization + head/tail window with an elision marker. Inputs too small to gain are returned unchanged with an honest header.

Framework: The Crush Rules

1. Crush between the tool and the context, not after: the token is spent the moment raw output enters the window — the crush step lives in the pipeline (tool | crush | context), not in cleanup. Retroactive crushing saves nothing already paid for.
2. Structural beats semantic for data: summarizing JSON with a model costs tokens, adds latency, and can hallucinate; schema+samples+stats is free, instant, and provably faithful — the numbers are computed, not paraphrased. Save model-summarization for prose, where structure can't do the work.
3. Errors are sacred: the log crusher's contract is that every error/warning line survives regardless of compression — a crush that can lose the one line that mattered is a corruption, not a compression. Any custom crushing keeps this invariant.
4. The escape hatch is part of the artifact: every crushed block states where the original lives ("full response in /tmp/response.json — fetch rows by id if needed"), because reversibility is what makes aggressive crushing safe.
5. Know when not to: non-uniform rows where each is signal, data being diffed byte-for-byte, legal/audit content, and anything under ~50 lines (the crush header costs more than it saves — the script says so itself). Crushing is a default for bulk, not a reflex for everything.

Output Format

Crushed: [payload] — ~[X] → ~[Y] tokens ([Z]% smaller)

[The crushed artifact, script header included]

Kept raw: [what wasn't crushed and why] · Original: [where it lives, how to fetch] Pipeline note: [where the crush step now sits in this workflow]

Quality Checks

• The token math appears — before, after, percent
• JSON crushes carry schema and computed stats, never paraphrased numbers
• Every error/warning line in a log crush survived
• The original's location and fetch route are stated
• Too-small inputs were returned unchanged, honestly

Anti-Patterns

• Do not summarize data with a model when structure can compress it — paraphrased numbers are hallucination surface
• Do not crush non-uniform, every-row-is-signal data — filter or keep it
• Do not drop the escape hatch — irreversible compression turns a saving into a gamble
• Do not crush after the tokens are spent — the step belongs in the pipeline
• Do not let the crush eat errors — the invariant outranks the ratio

Based On

The context-compression layer pattern — structural compression of tool outputs before the LLM (as in Headroom) — rebuilt here as a keyless, deterministic, stdlib skill.