Spec Craft

Spec Craft

LLM-judgment critique of spec quality (proposals + ADRs) against a curated rubric catalog from the spec-quality canon. Per-section critique with rubric-to-section mapping. Second member of the craft-pipeline initiative; highest-leverage craft skill because spec quality compounds across the entire planning → implementation → review lifecycle below it. Emits 3-axis findings (tier × impact × confidence per ADR 0019).

When to Use

• During PR review on a new or substantially-rewritten spec
• After authoring a proposal, before circulating it for review
• When onboarding a new contributor (audit specs they introduced)
• Periodically (per-sprint or per-release) to catch spec drift
• As the cross-cutting spec critic for harness-brainstorming (when newly-authored specs land)
• NOT for spec-structure enforcement (use harness-soundness-review — that's the rule-based floor)
• NOT for README / general-doc critique (use docs-craft #2)
• NOT for autofix / spec rewriting (this is judgment-only; v1.x may add align-spec sibling)
• NOT for source-code comment critique (use code-craft #4 or docs-craft #2)
• NOT for RFCs in v1 (v1.x)

Process

Phase 1: DISCOVER — Find spec files

1. Read project configuration. Check harness.config.json for:

   • craft.spec.enabled — gate (default true)
   • craft.spec.maxFiles — doc count cap (default 50)
   • craft.spec.maxSectionsPerFile — per-doc section cap (default 10)

2. Discover spec files:

   • proposals: docs/changes/*/proposal.md and docs/changes/*/<sub>/proposal.md (one level of nesting, for initiatives with sub-projects)
   • ADRs: docs/knowledge/decisions/*.md (excluding README)
   • Restrict via --kinds proposal or --kinds adr for single-kind runs.

Phase 2: PARSE — Split into named sections

For each spec file:

1. Strip YAML frontmatter (--- ... --- block at top, if present).
2. Split by H2 (## ...) into named sections. Each section captures:
   • heading — original H2 text (e.g., "Decisions", "Out-of-scope (v1)")
   • canonical — normalized form for rubric matching ("decisions", "out-of-scope-v1")
   • body — content between this H2 and the next
   • line / endLine — line range
3. Subsections (H3) stay part of the parent H2's body. v1.x may add per-H3 critique for sections like Decisions that have one row per H3.

Phase 3: CRITIQUE — Per (file, section, rubric) loop

7 seed rubrics:

Rubric	Title	Applies to
SPEC-R001	Sharpness vs vagueness	* (all sections)
SPEC-R002	Cuts at the joints	decisions, scope, technical-design
SPEC-R003	Two readers, same understanding	decisions, success-criteria
SPEC-R004	Load-bearing decision vs ambient context	decisions, overview
SPEC-R005	Honest rationalizations	rationalizations* (regex)
SPEC-R006	Non-goals are non-goals	out-of-scope*, non-goals* (regex)
SPEC-R007	Stranger in 6 months	* (all sections)

For each eligible (section, rubric) pair:

1. Build prompt with rubric description + spec file path + section heading + section body (truncated to 2000 chars if longer for cost control).
2. LLM returns fenced JSON: null (rubric doesn't apply / section is fine) OR { tier, impact, confidence, message }.
3. On non-null: emit a SpecFinding with cite.rubricId populated for ADR 0020 traceability.

Phase 4: REPORT — Aggregate + cost telemetry

Emit SpecCraftOutput:

{
  findings: SpecFinding[];
  summary: {
    phaseRun: ['critique'];
    durationMs: number;
    llmCalls: { provider, model, count, costUsd };
    catalog: { rubricsApplied: string[] };
    docsScanned: number;
    sectionsScanned: number;
    runId: string;
  }
}

Harness Integration

• harness spec-craft — CLI entry. --files <glob> / --kinds proposal,adr / --sections decisions,scope / --max-files <n> / --max-sections-per-file <n> / --json / --verbose.
• mcp__harness__spec_craft — MCP tool. Same input/output. Consumed by agents.
• Cross-cutting API: critiqueSpecFile(file, opts) exported from packages/cli/src/spec-craft/index.ts. Future craft skills (or harness-brainstorming) can call this when they have a doc in hand without re-walking the project.
• Shared craft infrastructure (extracted on this PR): LlmProvider, MockLlmProvider, derivePriority, 3-axis types all live in packages/cli/src/shared/craft/. design-craft + naming-craft + spec-craft import from there; design-craft + naming-craft keep their old import paths via re-export shims.

Success Criteria

See docs/changes/craft-pipeline/spec-craft/proposal.md for the full 34 success criteria. Highlights:

• 7 seed rubrics ship at catalog/rubrics/<id>.ts (file-per-rubric, matches naming-craft)
• 3-axis output preserved (tier × impact × confidence, never collapsed)
• cite.rubricId populated on every finding (ADR 0020)
• Section parser strips frontmatter; splits by H2 only
• Rubric-to-section mapping skips silently when rubric doesn't apply
• critiqueSpecFile cross-cutting API works on a single file without project walk
• All existing design-craft + naming-craft tests still pass after shared/craft extraction (zero behavior change)

Examples

Example: Vague Decisions section

Input: A proposal's ## Decisions section reads:

| Decision | Why |
|----------|-----|
| Use modern stack | scalable and clean |
| Defer auth | not in scope |

Output (mock LLM):

SPEC-R001 [polish/medium/medium] ## Decisions:34
  "Modern stack" and "scalable and clean" are vague — no concrete framework
  named, no metric for "scalable", no operational definition of "clean".
  Sharpen: name the framework, state the scale target (req/sec, team size),
  define what "clean" means in observable terms.
SPEC-R004 [polish/medium/medium] ## Decisions:34
  The Decisions section pads load-bearing choices with vague qualifiers
  rather than naming the trade-off chosen and the rejected alternative.

Example: Strawmanned Rationalizations

Input: A ## Rationalizations to reject section reads:

| "Use a different framework" | Other frameworks are worse |

Output:

SPEC-R005 [foundational/large/high] ## Rationalizations to reject:88
  "Other frameworks are worse" is a strawman — not stated charitably, not
  paired with a specific reason. Steelman the rejected position: name the
  competing framework, name its strongest feature, then explain the specific
  trade-off that made it unsuitable here.

Example: Empty project — no specs

Input: Project has no docs/changes/ or docs/knowledge/decisions/ directory.

Output:

No spec findings.

Summary: 0 findings across 0 docs (0 sections, 7 rubrics, 0 LLM calls, $0.0000, 3ms)

Gates

• No autofix. Sibling align-spec deferred until signal warrants safe-to-apply rewrites.
• No README / general doc critique. docs-craft (#2) territory.
• No source-code comment critique. code-craft / docs-craft territory.
• No B' bootstrap. Same posture as naming-craft v1.
• No graph persistence. Phase 1 MVP.
• No vision/deep mode. Specs are text.
• No structural floor enforcement. harness-soundness-review checks the floor; spec-craft assumes the floor is satisfied and critiques the ceiling.

Escalation

• When LLM cost is too high: drop maxSectionsPerFile to 5 or maxFiles to 25. Per-doc cost = sections × rubrics × per-call. Rubric-to-section mapping already prunes most calls; further: use --sections decisions to target the highest-value section.
• When a rubric produces high false-positive rate: v1 has no per-rubric disable; v1.x adds craft.spec.disabledRubrics: ['SPEC-R007']. Until then: filter findings by cite.rubricId in your consumer.
• When a spec has intentionally aspirational vagueness (e.g., a manifesto-style Overview): SPEC-R001 will flag it; low-confidence findings are de-emphasized per ADR 0019. v1.x adds per-section opt-out via <!-- spec-craft:skip --> HTML comment.
• When you want a doc-level summary instead of per-section findings: v1 is per-section only; v1.x adds a --mode doc opt-in for whole-doc critique.
• When you want to critique RFCs: v1.x. For now, point --files <rfc.md> at a single file — the section parser works on any markdown.

Status

v1 — in implementation. See:

• Spec: docs/changes/craft-pipeline/spec-craft/proposal.md
• Roadmap entry: craft-pipeline sub-project #6 (the highest-leverage craft skill)
• Sibling craft skills: harness-design-craft (design-pipeline #6), naming-craft (craft-pipeline #1)
• Shared infrastructure: packages/cli/src/shared/craft/ (extracted on this PR)
• Future: align-spec (FIX side), docs-craft (#2), test-craft (#3), code-craft (#4) — each can call critiqueSpecFile if they want spec-level critique for a doc they're already processing.