From nickcrew-claude-ctx-plugin
Validates claims in markdown docs against codebase reality by extracting assertions like file paths, commands, function refs, behavioral claims, and dependencies. Use after code changes, before releases, or when docs seem untrustworthy.
npx claudepluginhub nickcrew/claude-cortexThis skill uses the workspace's default tool permissions.
Verify that what documentation *says* is actually *true* by extracting testable claims
Audits documentation accuracy against code via multi-phase workflow: claim extraction, compilability checks, behavioral verification, and cross-doc consistency. Use for pre-release doc audits.
Audits documentation accuracy against code via multi-phase workflow: assessment, compilability checks, behavioral verification, consistency, and structure analysis. Use for verifying code examples, claims, or pre-release audits.
Detects documentation drift, stale references, and cross-document inconsistencies in projects. Scans code-doc mismatches, broken links, outdated versions, and git staleness.
Share bugs, ideas, or general feedback.
Verify that what documentation says is actually true by extracting testable claims
and checking them against the codebase. Complements doc-maintenance (which handles
structural health) by handling semantic accuracy.
doc-maintenance to go deeper than structural checks| Resource | Purpose | Load when |
|---|---|---|
scripts/extract_claims.py | Deterministic claim extraction from markdown | Always (Phase 1) |
scripts/verify_claims.py | Automated verification against codebase | Always (Phase 2) |
references/claim-taxonomy.md | Full taxonomy of claim types with examples | Triaging unclear claims |
Phase 1: Extract → Pull verifiable claims from docs (deterministic script)
Phase 2: Verify → Check claims against codebase (automated + AI)
Phase 3: Report → Classify failures by severity and type
Phase 4: Remediate → Fix or flag broken claims
Run the extraction script to parse all markdown files and pull out verifiable assertions:
python3 skills/doc-claim-validator/scripts/extract_claims.py [--json] [--root PATH] [--scope docs|manual|all]
The script extracts these claim types from markdown:
| Type | What it captures | Example in docs |
|---|---|---|
file_path | Inline code matching file path patterns | `src/auth/login.ts` |
command | Code blocks or inline code with shell commands | `npm run build` |
code_ref | Function, class, method references in inline code | `authenticate()` |
import | Import/require statements in code blocks | import { Router } from 'express' |
config | Configuration keys, env vars, settings | `MAX_RETRIES=3` |
url | External links (http/https) | [docs](https://example.com) |
dependency | Package/library name claims | "Uses Redis for caching" |
behavioral | Assertions about what code does | "The system retries 3 times" |
The first 6 types are extracted deterministically. The last 2 (dependency, behavioral) require
AI analysis and are handled in Phase 2.
Output: A structured list of claims with source file, line number, claim type, and the literal text of the claim.
Run the verification script on the extracted claims:
python3 skills/doc-claim-validator/scripts/verify_claims.py [--json] [--root PATH] [--claims-file PATH] [--check-staleness]
Pass --check-staleness to enable git-based drift analysis (see below).
The script checks each claim type differently:
| Claim type | Verification method | Pass condition |
|---|---|---|
file_path | os.path.exists() | File exists at referenced path |
command | shutil.which() + script check | Binary exists or script file exists |
code_ref | grep -r for function/class name | Symbol found in codebase |
import | Check module exists in project or deps | Module resolvable |
config | Grep for config key in source | Key found in config files or code |
url | HTTP HEAD request (optional, off by default) | Returns 2xx/3xx |
Pass --check-urls to enable URL verification (slow, requires network).
After the automated pass, dispatch haiku agents to verify claims the script cannot:
Agent 1 — Dependency claim verifier (subagent_type: "Explore", model: "haiku"):
Read package.json, requirements.txt, go.mod, Cargo.toml, or equivalent dependency
manifests. Cross-reference any doc claims about libraries, frameworks, or services used.
Report claims that reference dependencies not in the project.
Agent 2 — Behavioral claim verifier (subagent_type: "Explore", model: "haiku"):
For each behavioral claim (e.g., "retries 3 times", "caches for 5 minutes", "validates
input before processing"), find the relevant code and verify the claim is accurate.
Report mismatches between documented behavior and actual implementation.
Agent 3 — Code example verifier (subagent_type: "Explore", model: "haiku"):
For code blocks in docs that show usage examples, verify the function signatures,
parameter names, return types, and import paths match the current codebase. Report
examples that would fail if copy-pasted.
Launch all three agents in parallel.
For claims that pass existence checks, compute a drift score to surface likely-stale claims:
python3 skills/doc-claim-validator/scripts/verify_claims.py --check-staleness
For each passing claim, the script:
High-drift claims are the best candidates for AI review — the target changed heavily but the doc didn't, so the doc is probably describing outdated behavior.
The staleness report is appended as a ranked table, sorted by score descending.
Merge automated and AI findings into a single report. Classify each failed claim:
| Level | Meaning | Example |
|---|---|---|
| P0 | User-facing doc claims something that would break if followed | Tutorial shows deleted API endpoint |
| P1 | Dev doc references nonexistent code construct | README references auth.validate() which was renamed |
| P2 | Behavioral claim no longer accurate | "Retries 3 times" but retry logic was removed |
| P3 | Dependency/import claim outdated | "Uses Express" but migrated to Fastify |
| P4 | Minor inaccuracy, cosmetic | Config key renamed but behavior unchanged |
| Category | Description |
|---|---|
missing_target | Referenced file, function, or symbol doesn't exist |
wrong_signature | Function exists but signature differs from doc |
stale_behavior | Behavioral claim doesn't match implementation |
dead_dependency | Doc references a dependency not in the project |
broken_example | Code example would fail if executed |
dead_url | External link returns 4xx/5xx |
phantom_config | Config option referenced in docs doesn't exist in code |
For each failed claim, decide the action:
| Action | When | How |
|---|---|---|
| Update doc | Code is correct, doc is stale | Edit doc to match code |
| Flag for review | Unclear if code or doc is wrong | Create issue for human review |
| Remove claim | Referenced feature was deleted | Remove or rewrite section |
| Update example | Code example is outdated | Rewrite example against current code |
Route remediation to the appropriate agent per doc-maintenance conventions:
reference-builder for API/CLI reference docstechnical-writer for architecture and developer docslearning-guide for user-facing tutorials and guidesThis skill is designed to run after doc-maintenance:
doc-maintenance → Structural health (links, orphans, folders, staleness)
doc-claim-validator → Semantic accuracy (do claims match reality?)
The two skills share the same severity scale and remediation agent routing. Results from both can be combined into a single documentation health report.
`true` is not a file)docs/archive/) — they're historical<!-- no-verify --> commentscripts/extract_claims.py — Deterministic claim extraction from markdown filesscripts/verify_claims.py — Automated verification of extracted claims against codebasereferences/claim-taxonomy.md — Full taxonomy of claim types with extraction patterns and examples