quality-auditor

Quality Auditor (Comprehensive Audit + Verification + Metrics)

Purpose: Comprehensive quality audit against planning documents + quantitative metric tracking + verification discipline enforcement.

v3.0.0: Absorbed evaluation (metrics) and verification-before-completion (evidence discipline)

---

Absolute Prohibitions

1. Do not modify code directly — modification is the implementation agent's responsibility
2. Do not criticize without evidence — all feedback must cite documents under docs/planning/
3. Do not audit without planning documents — if both management/mini-prd.md and docs/planning/*.md are absent, run /governance-setup first

---

Immediate Actions on Trigger

1. Verify planning document existence (Mini-PRD or Socrates)
2. Load context (read reference documents)
3. Two-stage review (Spec Compliance → Code Quality)
4. DDD (Demo-Driven Development) validation
5. Security validation (invoke /security-review)
6. Dynamic validation (run tests)
7. UI/UX browser validation (agent-browser CLI + Lighthouse CLI)
8. Write quality report + provide fix guidance

---

Prerequisites (auto-checked at skill start)

Before starting the audit, verify the following when the skill is triggered.

1. Planning documents exist: at least one of management/mini-prd.md or docs/planning/*.md must be present.
   • If neither exists: "No planning documents found. Please run /governance-setup first."

---

Skeptical QA Baseline

Default assumption: "It doesn't work." Prove otherwise with evidence. (Inspired by Harness Evaluator pattern.)

Rule	Violation Blocked
All scores require evidence (file:line, test output, screenshot)	Score without evidence = 0
Console errors > 0 → Functionality capped at 7/10	Ignoring console errors
Uncaught exceptions → auto FAIL for that route	Rationalizing "minor" exceptions
Untestable (server down) → Score 0, not skip	"Couldn't test, so pass"
BLOCKING issues must be fixed before deploy	Shipping with known blockers

Issue Classification (applies to all audit output):

• BLOCKING — data loss, auth bypass, broken core flow, uncaught exception → must fix before deploy
• NON-BLOCKING — cosmetic issue, minor UX gap, non-critical warning → ship with ticket

---

Execution Process

Step 1: Check Planning Documents

# One of the two options
ls management/mini-prd.md 2>/dev/null        # Option A: Mini-PRD
ls docs/planning/*.md 2>/dev/null             # Option B: Socrates

Mini-PRD required fields: purpose, features, tech_stack Socrates required documents: 01-prd.md, 02-trd.md, 07-coding-convention.md

Step 2: Two-Stage Review

Stage 1: Spec Compliance

• Requirements match: are the PRD's core features implemented correctly?
• Missing features: are there any gaps in edge cases or error handling?
• YAGNI violations: are there unnecessary features not specified in the plan?

Stage 2: Code Quality

• SOLID/Clean Code: is the structure readable and extensible?
• Security: are there vulnerabilities such as exposed API keys or SQL injection?
• Performance: are there unnecessary re-renders or waterfall fetching?

Step 3: DDD Validation

• Demo pages: verify independent demo pages exist for each UI task
• Screenshot comparison: check that demos match mockups in design/
• Console integrity: confirm no browser console errors

Step 4: Security Validation

/security-review --path src --summary

Severity	Meaning	Deployable
CRITICAL	Immediate fix required	No — cannot deploy
HIGH	Fix recommended before deployment	Conditional
MEDIUM	Known issue	Yes — can deploy

Step 5: Dynamic Validation (Test Execution)

Project Type	Test Command
Node.js	npm test
Python	pytest
Python (Poetry)	poetry run pytest

Step 6: UI/UX Browser Validation (optional)

Uses agent-browser CLI or Lighthouse CLI

# 1. Open page + take snapshot
agent-browser open http://localhost:3000
agent-browser snapshot                    # accessibility tree (@ref based)
agent-browser screenshot audit.png        # visual capture

# 2. Lighthouse audit (accessibility + performance + SEO)
npx lighthouse http://localhost:3000 --output=json --quiet

# 3. Check console errors
agent-browser console                     # error/warning count

AI Slop Detection — Auto-deduct from visual score: ≥1 pattern → −1pt, ≥3 patterns → −2pt.

#	Pattern	#	Pattern
1	Hero section with no real image (placeholder/gradient)	6	Generic icons only (no custom illustrations)
2	3-column generic feature grid	7	Empty state not handled
3	Meaningless gradient decoration	8	Hardcoded demo data visible in UI
4	Lorem ipsum text remaining	9	Excessive rounded-xl on everything
5	Identical card layout repeated throughout	10	Purposeless animation/motion

---

Audit Result Submission

Quality Summary

┌─────────────────────────────────────────┐
│ Quality Audit Result                    │
├─────────────────────────────────────────┤
│ Score: 85/100                           │
│ Verdict: CAUTION                        │
│                                         │
│ ✅ Feature conformance: 95%             │
│ ⚠️  Conventions: 75%                   │
│ Security: 88% (1 medium issue)          │
│ ✅ Tests: passed (coverage 82%)         │
└─────────────────────────────────────────┘

Verdict criteria:

Score	Verdict	Meaning
90+	PASS	Ready to deploy immediately
70–89	CAUTION	Deploy after minor fixes
Below 70	FAIL	Major fixes required

Critical Defects

BLOCKING	Priority	Category	Description	Related File
✅ YES	Critical	Security	Hardcoded API key	src/api/auth.py:23
✅ YES	High	Bug	Missing duplicate check	src/api/auth.py:45

---

Skill Integration

Audit Result	Recommended Skill
Spec mismatch	/agile iterate
Code quality issues	/checkpoint → re-audit
Security vulnerabilities	Re-run /security-review
Deep review needed	/multi-ai-review

Agent Team Integration (project-team)

Automates the deployment approval process in collaboration with the QA Manager agent.

/audit → calculate quality score → request QA Manager approval
    ↓
✅ Approved → proceed to deployment
⚠️  Conditional → re-validate after fixing issues
❌ Rejected → send feedback to Specialist

Detailed integration patterns: see references/agent-integration.md

---

Step 7: Verification Discipline (Iron Law)

Absorbed from verification-before-completion. Applies to ALL completion claims.

Iron Law: No claims without fresh evidence.

Before asserting any state ("tests pass", "bug fixed", "build succeeds"):

1. IDENTIFY — What command proves this claim?
2. RUN — Execute the full command (fresh, complete)
3. READ — Check full output, exit code, failure count
4. VERIFY — Does the output confirm the claim?
   - NO → state actual status with evidence
   - YES → state claim with evidence
5. ONLY THEN — Make the claim

Claim	Required evidence	NOT sufficient
Tests pass	Test command output: 0 failures	Previous run, "should pass"
Lint clean	Lint output: 0 errors	Partial check, inference
Build succeeds	Build command: exit 0	Lint passed, logs look OK
Bug fixed	Original symptom test: passes	Code changed, assumed fixed

Red flags — stop immediately if you catch yourself:

• Using "should", "probably", "seems to"
• Expressing satisfaction before running verification
• Trusting agent success reports without checking

---

Step 8: Quantitative Metrics (Dashboard)

Absorbed from evaluation. Optional — run when metrics tracking is needed.

Code Quality Metrics

Metric	Command	Target	Warning
Test coverage	pytest --cov / vitest --coverage	≥70%	<60%
Cyclomatic complexity	radon cc / eslint complexity	≤10	>15
Code duplication	jscpd / pylint duplicate	≤5%	>10%
Lint errors	ruff / eslint	0	>5
Type errors	mypy / tsc	0	>0
Security score	bandit / npm audit	0 critical	any critical

Agent Performance Metrics

Metric	Target
Task completion rate	≥95%
Average retries per task	≤2
First-attempt success rate	≥80%

Store metrics in .claude/metrics/ for trend tracking across phases.

---

Reference Documents

• references/agent-integration.md — QA Manager integration patterns, feedback routing

---

Last Updated: 2026-04-01 (v3.1.0 — Skeptical QA baseline + BLOCKING/NON-BLOCKING classification + AI Slop Detection)

---

Goal-backward Verification (GSD 패턴)

핵심 원칙: Task 완료 ≠ Goal 달성

검증 흐름

Goal (목표)
  ↓
Must-have (참이어야 할 것)
  ↓
Must-exist (존재해야 할 것)
  ↓
Must-wired (연결되어야 할 것)
  ↓
실제 코드베이스 검증

Step 1: Goal 파악

# TASKS.md에서 현재 Phase/작업의 목표 추출
GOAL=$(grep -A5 "## Phase" TASKS.md | grep -E "^>" | head -1)

Step 2: Must-haves 도출

목표에서 역산하여 필수 조건 도출:

Goal: "사용자가 채팅할 수 있어야 함"
  ↓
Must-have:
  - 메시지 전송 가능
  - 메시지 수신 가능
  - 메시지 표시 가능

Step 3: Must-exist 확인

각 must-have에 대해 실제 코드 존재 확인:

# 예: 채팅 기능 검증
grep -r "sendMessage\\|ChatInput\\|MessageList" src/

Step 4: Must-wired 확인

컴포넌트 간 연결 확인:

# import/export 관계 확인
grep -r "import.*from.*chat" src/

Step 5: VERIFICATION.md 생성

# {Phase} - Verification Report

**검증일:** {date}
**상태:** {PASS|FAIL}

## Goal
{검증한 목표}

## Must-haves 검증

| ID | Must-have | Status | Evidence |
|----|-----------|--------|----------|
| M-01 | {항목} | ✅/❌ | {파일:라인} |

## Gaps (실패 시)
- {누락된 항목}
- {수정 필요 사항}

## 다음 단계
{PASS 시: 다음 Phase}
{FAIL 시: Gap 해결 작업}

검증 실패 처리

1. Gaps 목록화
2. TASKS.md에 수정 작업 추가
3. 수정 후 재검증