Validation Runner — ultrathink

Overview

Execute the full project validation suite — TypeScript compilation, test runner, linter, formatter — and produce a structured JSON report. This feeds REAL DATA into agent evaluations instead of opinion-based assessment. Run before an audit (baseline) and after fixes (delta) to close the proof loop.

Prerequisites

Before running, verify:

• Project root: !ls package.json 2>/dev/null || echo "NO package.json — cannot detect stack"
• TypeScript config: !ls tsconfig.json 2>/dev/null || echo "NO tsconfig.json — skip type checking"
• Node modules: !ls node_modules/.package-lock.json 2>/dev/null || echo "NO node_modules — run npm install first"
• BoardClaude state dir: !ls .boardclaude/ 2>/dev/null || echo "NOT FOUND — will create"
• Previous validation: !ls .boardclaude/validation/latest.json 2>/dev/null || echo "None — this is the first run"

Steps

1. Detect project stack by reading package.json:

   • Check dependencies and devDependencies for framework (next, react, express, etc.)
   • Check for TypeScript (typescript in devDependencies, tsconfig.json exists)
   • Check for test runner (jest, vitest, mocha, etc.)
   • Check for linter (eslint, next lint script)
   • Check for formatter (prettier)
   • Record detected stack in results

2. Ensure .boardclaude/validation/ directory exists. Create it if missing.

3. If a previous latest.json exists, archive it to .boardclaude/validation/run-{timestamp}.json before overwriting. This preserves history for delta tracking.

4. Run TypeScript compilation check:

   npx tsc --noEmit 2>&1
   

   • Parse output: count lines matching error TS\d+ for error count
   • Count lines matching warning for warning count
   • Capture up to 10 specific error messages with file paths and line numbers
   • If tsconfig.json is missing, record "skipped": true and move on

5. Run test suite:

   npm test -- --reporter=json 2>&1
   

   • If Jest: parse JSON output for numPassedTests, numFailedTests, numTotalTests
   • If Vitest: parse JSON reporter output similarly
   • Extract coverage percentage if available (--coverage flag or existing config)
   • If no test script configured in package.json, record "skipped": true
   • Capture names of failing tests (up to 10)

6. Run linter:

   npx next lint --format=json 2>&1
   

   • If not a Next.js project, fall back to npx eslint . --format=json 2>&1
   • Parse JSON output: count errors and warnings
   • Capture up to 10 specific lint issues with file paths
   • If no eslint config found, record "skipped": true

7. Run formatter check:

   npx prettier --check . 2>&1
   

   • Count total files checked vs files needing formatting
   • Calculate compliance percentage: (checked - failing) / checked * 100
   • If prettier is not installed, record "skipped": true

8. Run Lighthouse audit (optional, only if available):

   npx lighthouse http://localhost:3000 --output=json --chrome-flags="--headless" 2>&1
   

   • Only attempt if a dev server is running (check port 3000)
   • Extract performance, accessibility, best-practices, SEO scores
   • If unavailable or dev server not running, record null for all fields

9. Assemble results into the structured JSON format (see Output Format below).

10. Save results to .boardclaude/validation/latest.json.

11. Calculate delta if a previous run exists:

    • Compare error counts, test counts, lint counts, format compliance
    • Produce a delta object showing changes: { "typescript_errors": -3, "tests_passed": +2, ... }
    • Include delta in the saved results

12. Report summary to the user:

    • One-line verdict: "12 type errors, 45/47 tests passing, lint score 94%, format 100%"
    • If delta exists: "Delta: -3 type errors, +2 tests passing, lint +1%"
    • Path to full results file

Output Format

Save to .boardclaude/validation/latest.json:

{
  "timestamp": "<ISO 8601>",
  "stack": {
    "framework": "next",
    "language": "typescript",
    "test_runner": "jest",
    "linter": "eslint",
    "formatter": "prettier"
  },
  "typescript": {
    "skipped": false,
    "errors": 0,
    "warnings": 0,
    "details": [
      { "file": "src/app/page.tsx", "line": 12, "code": "TS2345", "message": "..." }
    ]
  },
  "tests": {
    "skipped": false,
    "total": 47,
    "passed": 45,
    "failed": 2,
    "coverage": 78.5,
    "failing_tests": ["test name 1", "test name 2"]
  },
  "lint": {
    "skipped": false,
    "errors": 1,
    "warnings": 5,
    "details": [
      { "file": "src/lib/utils.ts", "line": 8, "rule": "no-unused-vars", "severity": "error" }
    ]
  },
  "format": {
    "skipped": false,
    "compliant_pct": 100,
    "files_checked": 42,
    "files_failing": 0
  },
  "lighthouse": {
    "performance": null,
    "accessibility": null,
    "best_practices": null,
    "seo": null
  },
  "delta": {
    "previous_timestamp": "<ISO 8601 or null>",
    "typescript_errors": 0,
    "tests_passed": 0,
    "tests_failed": 0,
    "lint_errors": 0,
    "lint_warnings": 0,
    "format_compliant_pct": 0
  },
  "summary": "12 type errors, 45/47 tests passing, 1 lint error, format 100%"
}

Integration with Audit Pipeline

When audit-runner invokes validation-runner before spawning agents:

1. Run validation-runner to produce latest.json
2. Read latest.json and inject summary into each agent's evaluation prompt:

   "Validation context: 12 type errors (details: ...), 45/47 tests passing (2 failing: ...), 1 lint error, format compliant."

3. Agents now have real data to cite — Boris can reference actual type errors, Lydia can verify patterns against real lint output
4. After audit and fix cycle, re-run validation-runner to measure improvement

Error Handling

• package.json missing: Abort with clear message — "No package.json found. Run from project root."
• node_modules missing: Warn and suggest npm install, then attempt to continue (tools may fail)
• TypeScript not installed: Skip type checking, set typescript.skipped = true
• No test script: Skip tests, set tests.skipped = true — do NOT fail the whole run
• Test suite fails (exit code 1): This is expected when tests fail. Parse the output normally — a non-zero exit code means there are failing tests, not that the runner broke.
• Linter not configured: Skip linting, set lint.skipped = true
• Prettier not installed: Skip format check, set format.skipped = true
• Command timeout: Set 60-second timeout per command. If exceeded, record "timeout": true for that check.
• JSON parse failure: If a tool outputs non-JSON, fall back to regex parsing of plain text output
• Lighthouse unavailable: Always optional. Record null values silently.