Validate test coverage for increment tasks, check AC-ID coverage, report test status, and verify coverage targets (NEW format with embedded tests in tasks.md)
Validates test coverage from tasks.md, checks AC-ID coverage, and reports test status for increment tasks. Use this to verify coverage targets and identify gaps before PR or increment closure.
/plugin marketplace add anton-abyzov/specweave/plugin install sw@specweavePurpose: Validate test coverage from tasks.md (NEW format with embedded tests)
Replaces: validate-coverage (old command for tests.md format)
Shows:
# Check tests for current increment
/sw:check-tests
# Check tests for specific increment
/sw:check-tests 0007
# Detailed report with test case breakdown
/sw:check-tests 0007 --detailed
# AC-ID coverage report only
/sw:check-tests 0007 --ac-coverage
OLD Format (pre-v0.7.0):
NEW Format (v0.7.0+):
Why the change?
/sw:check-tests 0007
📊 Test Status Report: 0007-smart-increment-discipline
═══════════════════════════════════════════════════════════════
Task Coverage Analysis:
✅ T-001: Update plan.md (N/A - documentation)
Validation: Manual review complete
✅ T-003: Create test-aware-planner structure (2 test cases, 85% coverage)
- Manual: Directory structure verified ✅
- Integration: tests/integration/test-aware-planner/agent-invocation.test.ts
• testAgentCanBeInvoked() ✅ (passed)
• testAgentReadsSpecPlan() ✅ (passed)
Coverage: 87% (target: 85%) ✅
⚠️ T-007: Create check-tests command (0 test cases, 0% coverage)
WARNING: No test plan defined!
Action: Add test cases to T-007 or mark as non-testable
❌ T-008: Implement check-tests logic (3 test cases, 45% coverage)
- Unit: tests/unit/check-tests.test.ts
• testParseTasksMd() ❌ (failing: Expected 3 tasks, got 2)
• testExtractTestCases() ✅ (passed)
• testCalculateCoverage() ⏸️ (not implemented)
Coverage: 45% ❌ (target: 85%, below threshold!)
Action: Fix failing test, implement missing test
═══════════════════════════════════════════════════════════════
Overall Coverage: 72% ⚠️ (target: 80-90%)
Breakdown:
- Testable tasks: 15/24 (62%)
- Non-testable tasks: 9/24 (38%) ✅
- Average coverage (testable): 78% ⚠️
- Tests passing: 12/18 (67%) ⚠️
- Tests failing: 3/18 (17%) ❌
- Tests not implemented: 3/18 (17%) ⏸️
═══════════════════════════════════════════════════════════════
Acceptance Criteria Coverage:
✅ AC-US1-01: Covered by T-001, T-003 (6 tests, 88% coverage)
✅ AC-US1-02: Covered by T-002, T-004 (4 tests, 92% coverage)
⚠️ AC-US2-01: Partially covered by T-007 (0 tests)
Missing: Unit tests for command logic
❌ AC-US3-05: Not covered by any task
Action: Add task for AC-US3-05 or mark as won't-do
═══════════════════════════════════════════════════════════════
🎯 Recommendations:
Priority: P1 (blocking issues)
1. Fix failing test in T-008: testParseTasksMd()
2. Implement missing test in T-008: testCalculateCoverage()
3. Add test plan to T-007 or mark as non-testable
Priority: P2 (coverage gaps)
4. Increase coverage in T-008 to ≥85%
5. Add unit tests for AC-US2-01
6. Create task for uncovered AC-US3-05
═══════════════════════════════════════════════════════════════
Summary:
- Status: ⚠️ NEEDS WORK (coverage below 80%, failing tests)
- Next: Fix failing tests, implement missing tests
- Run: npm test (to re-run all tests)
/sw:check-tests 0007 --detailed
[Shows expanded report with:]
- Full test case breakdown per task
- Code coverage metrics per file
- Test execution times
- Test file paths and line numbers
- Suggestions for improving coverage
/sw:check-tests 0007 --ac-coverage
═══════════════════════════════════════════════════════════════
Acceptance Criteria Coverage Report: 0007
✅ AC-US1-01: User can login with email/password
Covered by: T-001, T-003
Test cases: 6 (unit: 3, integration: 2, E2E: 1)
Coverage: 88%
Status: ✅ PASSING
✅ AC-US1-02: User can logout
Covered by: T-002, T-004
Test cases: 4 (unit: 2, integration: 2)
Coverage: 92%
Status: ✅ PASSING
⚠️ AC-US2-01: Command validates test coverage
Covered by: T-007, T-008
Test cases: 3 (unit: 3)
Coverage: 45% ⚠️
Status: ⚠️ PARTIAL (1 failing, 1 not implemented)
❌ AC-US3-05: Documentation is complete
Covered by: NONE
Test cases: 0
Coverage: 0%
Status: ❌ NOT COVERED
═══════════════════════════════════════════════════════════════
Summary:
- Total AC-IDs: 4
- Fully covered: 2 (50%)
- Partially covered: 1 (25%)
- Not covered: 1 (25%)
- Overall AC coverage: 75% ⚠️
For each task in tasks.md:
Testable tasks:
Non-testable tasks:
Priority-based action items:
// Pseudo-code for check-tests logic
const tasks = parseTasksMd('.specweave/increments/0007/tasks.md');
for (const task of tasks) {
if (task.testPlan === 'N/A') {
// Non-testable task
reportValidation(task);
} else {
// Testable task
const testCases = extractTestCases(task);
const results = runTests(testCases);
const coverage = calculateCoverage(results);
reportTestStatus(task, testCases, results, coverage);
}
}
From each task's "Test Cases" section:
**Test Cases**:
1. **Unit**: `tests/unit/auth.test.ts`
- testValidLogin(): ...
- testInvalidLogin(): ...
Extracts:
# For each test file, run:
npm test tests/unit/auth.test.ts
# Capture:
# - Pass/fail status
# - Error messages
# - Code coverage %
Actual coverage: 87%
Target coverage: 85%
Status: ✅ PASSING (actual ≥ target)
# From spec.md:
AC-US1-01: User can login
# From tasks.md:
T-001: Implement login
Acceptance Criteria: AC-US1-01
# Result:
AC-US1-01 → Covered by T-001 ✅
If increment has tests.md (pre-v0.7.0):
/sw:check-tests 0003 # Old format increment
⚠️ OLD FORMAT DETECTED: tests.md exists
This increment uses the old format (separate tests.md).
The check-tests command supports both formats.
[Parses tests.md with TC-IDs instead of tasks.md]
📊 Test Status Report: 0003 (OLD FORMAT)
Test Case Coverage:
✅ TC-001: User can login (3 tests, 90% coverage)
...
Detection logic:
const hasTestsMd = fs.existsSync('.specweave/increments/0003/tests.md');
if (hasTestsMd) {
// OLD FORMAT: Parse tests.md for TC-IDs
return checkTestsOldFormat(incrementId);
} else {
// NEW FORMAT: Parse tasks.md for embedded tests
return checkTestsNewFormat(incrementId);
}
When to run check-tests:
During development: After implementing tasks
# Implement T-001
# Write tests
/sw:check-tests # Verify tests pass
Before PR: Ensure coverage meets targets
/sw:check-tests 0007
# Fix any failing tests
# Commit and push
In CI/CD: Automated test validation
# In .github/workflows/test.yml
- name: Check test coverage
run: npx specweave check-tests $INCREMENT_ID
PM Gate 2: Part of increment closure
/sw:done 0007
# PM Agent runs /sw:check-tests automatically
# Verifies >80% coverage before closure
For CI/CD integration:
/sw:check-tests 0007
echo $? # Exit code
# Exit codes:
# 0 = All tests passing, coverage ≥80%
# 1 = Tests failing
# 2 = Coverage <80%
# 3 = AC-IDs not covered
# 4 = Command error (invalid increment, etc.)
The command reads:
.specweave/increments/{id}/tasks.md (NEW format).specweave/increments/{id}/tests.md (OLD format, if exists).specweave/increments/{id}/spec.md (for AC-IDs)/validate 0007 - Full increment validation (tasks, tests, docs)/progress - Current increment progress with test status/done 0007 - Close increment (runs check-tests automatically)| Aspect | OLD (validate-coverage) | NEW (check-tests) |
|---|---|---|
| Reads | tests.md | tasks.md |
| Test IDs | TC-001, TC-002 | Function names |
| Format | Separate file | Embedded in tasks |
| BDD | No | Yes (Given/When/Then) |
| AC Traceability | Manual | Automatic (AC-IDs) |
| Sync Issues | tasks ↔ tests | None (single file) |
| Coverage Target | Per test case | Per task + overall |
| Status | Deprecated (v0.7+) | Current |
For developers working on check-tests logic (T-008):
### T-001: Implement Feature
**Test Plan**:
- **Given** precondition
- **When** action
- **Then** outcome
**Test Cases**:
1. **Unit**: `tests/unit/feature.test.ts`
- testFeatureSuccess(): ...
- **Coverage Target**: 90%
**Overall Coverage Target**: 85%
interface TaskTest {
taskId: string;
testPlan: TestPlan | 'N/A';
testCases: TestCase[];
overallTarget: number;
}
interface TestPlan {
given: string;
when: string;
then: string;
and?: string;
}
interface TestCase {
level: 'Unit' | 'Integration' | 'E2E';
filePath: string;
functions: string[];
target: number;
}
async function runTest(testCase: TestCase): Promise<TestResult> {
// Run: npm test {filePath}
// Parse output for pass/fail
// Extract coverage %
return {
passed: boolean,
coverage: number,
errors: string[]
};
}
function generateReport(results: TaskTest[]): Report {
return {
taskCoverage: calculateTaskCoverage(results),
acCoverage: calculateACCoverage(results, specMd),
recommendations: generateRecommendations(results),
overallStatus: determineStatus(results)
};
}
check-tests is the NEW command (v0.7.0+) for validating test coverage from tasks.md with embedded test plans.
Key Features:
Replaces: validate-coverage (deprecated)
See also: ARCHITECTURE-PIVOT.md (rationale for NEW format)