fact-check

Fact Check

Verify the factual accuracy of a document against the actual codebase and git history. Extracts verifiable claims, checks each against source, corrects inaccuracies in place, and adds a verification summary.

This is not a re-review. It does not second-guess analysis, opinions, or design judgments. It does not change the document's structure or organization. It is a fact-checker — it verifies that the data presented matches reality, corrects what doesn't, and leaves everything else alone.

Instructions

Target File Detection

Determine what to verify from $1:

1. Explicit path: Verify that specific file (.html, .md, or any text document)
   • Resolve relative paths against cwd
2. No argument: Auto-detect the most recent HTML report:

   ls -t ${CLAUDE_PLUGIN_DATA}/reports/*.html | head -1
   

   If no reports found, inform the user and stop.

Document type detection — auto-detect from page content to adjust verification strategy:

Document Type	Detection	Verification Focus
diff-visual report	Contains "Diff Visual" in title/heading	Verify against the git ref the review was based on
plan-visual report	Contains "Plan Visual" in title/heading	Verify file references, names, architecture claims
project-recap report	Contains "Project Recap" in title/heading	Re-run git commands, verify activity narrative
agent-extension-visual report	Contains plugin analysis markers	Verify plugin structure, file paths, feature descriptions
Markdown document	.md extension	Verify file references, function/type names, behavior descriptions
Other	Fallback	Extract and verify whatever factual claims about code it contains

Language Detection

Determine the output language for the verification summary:

1. Explicit argument: --lang <code> (e.g., --lang ko, --lang fr, --lang zh) → use that language. Any language code is valid
2. User message text: Detect the language of the message (excluding path) and match it
   • Examples: Korean text → Korean, Japanese text → Japanese, "en español" → Spanish, "auf Deutsch" → German
3. Document language: Match the language of the document being verified
4. Default: English

Feedback File Detection

After determining the target file, check for a companion feedback.json:

1. Explicit argument: --feedback path/to/feedback.json
2. Auto-detect: Check ~/Downloads/feedback.json (macOS default download location) — verify report_path matches the target file. If multiple feedback*.json exist (e.g., feedback (1).json), use the most recent one.
3. No feedback: Proceed with standard full verification

When feedback.json is present, adjust verification strategy:

• Sections with status "issue" + feedback text: These are the user's primary concerns. Verify these sections FIRST and with extra scrutiny. The feedback text describes the specific problem — use it to guide what to check.
• Sections with status "ok": User reviewed and approved. Still verify, but at lower priority — only check quantitative claims and names.
• Sections with status "not-reviewed": Standard verification.

In the Phase 5 Report, include feedback-driven summary:

Feedback-guided verification:
  {N} sections flagged by user
  {N} issues confirmed and corrected
  {N} issues not reproduced (user concern was unfounded)

Phase 1: Extract Claims

Why: Systematic extraction prevents cherry-picking. Every verifiable claim must be identified before verification begins.

Read the target file. Extract every verifiable factual claim into 5 categories:

1. Quantitative: Line counts, file counts, function counts, module counts, test counts, any numeric metrics
2. Naming: Function names, type names, module names, file paths referenced in the document
3. Behavioral: Descriptions of what code does, how things work, before/after comparisons
4. Structural: Architecture claims, dependency relationships, import chains, module boundaries
5. Temporal: Git history claims, commit attributions, timeline entries

Skip subjective analysis: opinions, design judgments, readability assessments, severity ratings, recommendations. These aren't verifiable facts.

Phase 2: Verify Against Source

Why: Each claim category requires a different verification method. Using the wrong method (e.g., Grep for quantitative claims) produces false confirmations.

For each extracted claim, go to the actual source:

Naming claims — Glob + Read:

• Verify every file path exists
• Verify every function name, type name, and module name exists at the claimed location
• Check for typos, renames, or stale references

Quantitative claims — Bash git commands:

• Re-run git diff --stat, git log, git diff --name-status and compare output against the document's numbers
• Verify line counts with wc -l
• Verify file counts with Glob

Behavioral claims — Read source files:

• Read every file referenced and check function signatures, type definitions
• For diff-reviews: read both the ref version (git show <ref>:file) and working tree version to verify before/after claims
• Check that described behaviors match actual code logic

Structural claims — Grep + Read:

• Verify import/dependency relationships
• Check that architecture descriptions match actual module boundaries
• Verify that claimed connections between modules exist

Temporal claims — Git commands:

• Re-run git log commands to verify activity narrative
• Verify commit hashes, authors, dates, and messages
• Check that timeline entries match actual git history

Classify each claim:

• Confirmed: Claim matches the code/output exactly
• Corrected: Claim was inaccurate — note what was wrong and what the correct value is
• Unverifiable: Claim can't be checked (e.g., references a file that doesn't exist, or requires runtime testing)

Phase 3: Correct In Place

Why: Surgical corrections preserve the document's structure and style. Over-editing risks breaking HTML layout or changing the author's voice.

Use the Edit tool for surgical corrections:

Do correct:

• Incorrect numbers (line counts, file counts, commit counts)
• Wrong function names, type names, file paths
• Inaccurate behavior descriptions
• Swapped before/after comparisons
• Wrong git hashes, dates, or attributions
• Factual errors in Mermaid diagram node labels or edge descriptions

Do NOT change:

• HTML layout, CSS, or animations
• Document structure or section organization
• Mermaid diagram styling or layout (only fix factual labels/edges)
• Subjective analysis, opinions, or design judgments
• Writing style or tone

If a section contains a factual error, fix only the factual part. If a section is fundamentally wrong (not just a detail error), rewrite that section's content while preserving the surrounding HTML/markdown structure.

Phase 4: Add Verification Summary

Why: Transparency — readers can see what was checked, what changed, and what couldn't be verified.

Insert a verification summary into the document.

For HTML files — insert a verification section matching the page's existing design:

<section id="verification-summary" class="ve-card" style="--i: {next-index}">
  <h2>Verification Summary</h2>
  <div class="kpi-grid">
    <div class="kpi-card kpi-card--info">
      <span class="kpi-value">{total}</span>
      <span class="kpi-label">Claims Checked</span>
    </div>
    <div class="kpi-card kpi-card--success">
      <span class="kpi-value">{confirmed}</span>
      <span class="kpi-label">Confirmed</span>
    </div>
    <div class="kpi-card kpi-card--danger">
      <span class="kpi-value">{corrected}</span>
      <span class="kpi-label">Corrected</span>
    </div>
    <div class="kpi-card kpi-card--warning">
      <span class="kpi-value">{unverifiable}</span>
      <span class="kpi-label">Unverifiable</span>
    </div>
  </div>
  <details>
    <summary>Corrections Made</summary>
    <ul>
      <li>{description of each correction with file:line reference}</li>
    </ul>
  </details>
  <details>
    <summary>Unverifiable Claims</summary>
    <ul>
      <li>{claim that could not be verified and why}</li>
    </ul>
  </details>
</section>

Place the verification section as the last content section, before </main> or the closing layout wrapper.

For Markdown files — append at the end:

## Verification Summary

| Metric | Count |
|--------|-------|
| Claims Checked | {total} |
| Confirmed | {confirmed} |
| Corrected | {corrected} |
| Unverifiable | {unverifiable} |

### Corrections Made
- {description of each correction}

### Unverifiable Claims
- {claim and reason}

Gotchas

• Over-correcting opinions as facts: "This architecture is well-designed" is a subjective judgment, not a factual claim. Only correct things that can be verified against source — names, numbers, behaviors, paths. When in doubt, skip it.
• Modifying HTML structure: The Edit tool is for surgical text corrections only. Do not reorganize sections, move content between sections, or change HTML wrapper elements. If a section is fundamentally wrong, rewrite the text content inside the existing <section> tags.
• Stale git refs in diff-visual reports: A diff-visual report captures a snapshot. If new commits landed since the report was generated, the fact-checker sees different data than the report author did. Verify against the same ref the report was based on (look for commit hashes in the report), not HEAD.
• Feedback.json from wrong report: The auto-detect checks ~/Downloads/feedback.json which may be from a completely different report. Always verify the report_path field matches the target file before using feedback data.
• Counting claims too aggressively: Not every number in a report is a "claim" worth verifying. Focus on claims that matter — metrics in KPI cards, file counts in summaries, function names in architecture descriptions. Ignore incidental numbers in prose.
• Mermaid diagram labels: Mermaid node labels that contain function or file names are factual claims. If a diagram says validateAuth() but the actual function is verifyAuth(), that's a correction. But don't change diagram layout or styling.

Phase 5: Report

Output a summary to the user:

Fact-check complete: {file path}

  {total} claims checked
  {confirmed} confirmed
  {corrected} corrected
  {unverifiable} unverifiable

{If corrections were made, list the top 3-5 most significant corrections}
{If nothing needed correction, note that verification confirms accuracy}