runtime-verification

Runtime Verification

This skill verifies that an implementation actually works at runtime, not just that it compiles and passes lint/tests.

Purpose

Quality checks (lint, test, typecheck) answer "does it compile?" This skill answers "does it work?" by:

• Starting dev servers and verifying they respond
• Running smoke tests against new/modified endpoints
• Executing E2E test suites if available
• Visually verifying UI changes in the browser
• Verifying acceptance criteria programmatically

Read Timeout Settings

Before running verifications, read timeout configuration (local > project > user > defaults):

DEV_STARTUP_TIMEOUT=$(jq -r '.timeouts.devServerStartup // empty' .claude/settings.gh-workflow.local.json 2>/dev/null)
[ -z "$DEV_STARTUP_TIMEOUT" ] && DEV_STARTUP_TIMEOUT=$(jq -r '.timeouts.devServerStartup // empty' .claude/settings.gh-workflow.json 2>/dev/null)
[ -z "$DEV_STARTUP_TIMEOUT" ] && DEV_STARTUP_TIMEOUT=$(jq -r '.timeouts.devServerStartup // empty' "$HOME/.claude/settings.gh-workflow.json" 2>/dev/null)
[ -z "$DEV_STARTUP_TIMEOUT" ] && DEV_STARTUP_TIMEOUT="30"

E2E_TIMEOUT=$(jq -r '.timeouts.e2eTest // empty' .claude/settings.gh-workflow.local.json 2>/dev/null)
[ -z "$E2E_TIMEOUT" ] && E2E_TIMEOUT=$(jq -r '.timeouts.e2eTest // empty' .claude/settings.gh-workflow.json 2>/dev/null)
[ -z "$E2E_TIMEOUT" ] && E2E_TIMEOUT=$(jq -r '.timeouts.e2eTest // empty' "$HOME/.claude/settings.gh-workflow.json" 2>/dev/null)
[ -z "$E2E_TIMEOUT" ] && E2E_TIMEOUT="120"

VERIFICATION_TIMEOUT=$(jq -r '.timeouts.verificationScript // empty' .claude/settings.gh-workflow.local.json 2>/dev/null)
[ -z "$VERIFICATION_TIMEOUT" ] && VERIFICATION_TIMEOUT=$(jq -r '.timeouts.verificationScript // empty' .claude/settings.gh-workflow.json 2>/dev/null)
[ -z "$VERIFICATION_TIMEOUT" ] && VERIFICATION_TIMEOUT=$(jq -r '.timeouts.verificationScript // empty' "$HOME/.claude/settings.gh-workflow.json" 2>/dev/null)
[ -z "$VERIFICATION_TIMEOUT" ] && VERIFICATION_TIMEOUT="180"

Quick Verification Fast Path

Before running the full discovery process, check for an existing verification script. Many mature projects already wire everything up in one command — if one exists, run it and skip the rest.

# Check for dedicated verification scripts
ls verify.sh test-e2e.sh smoke-test.sh scripts/verify* 2>/dev/null

# Check Makefile for verify/e2e/smoke targets
grep -E "^(verify|e2e|smoke|integration-test):" Makefile 2>/dev/null

If found, run it with a timeout:

timeout $VERIFICATION_TIMEOUT ./verify.sh 2>&1  # or make verify, etc.

If it passes, skip to Output Format. If it fails or no script exists, continue with full discovery.

Discovery Process

Step 1: Check CLAUDE.md for Dev/Test Commands

Search project instructions for any development or verification commands:

grep -iE "(dev server|npm run|yarn |pnpm |python.*run|go run|cargo run|docker.compose|make\s+\w+|uvicorn|gunicorn|flask run)" .claude/CLAUDE.md CLAUDE.md 2>/dev/null
grep -iE "(verify|e2e|smoke|integration|acceptance)" .claude/CLAUDE.md CLAUDE.md 2>/dev/null

Step 2: Check for E2E Test Frameworks

# Playwright
ls playwright.config.* 2>/dev/null
grep -l "playwright" package.json 2>/dev/null

# Cypress
ls cypress.config.* cypress/ 2>/dev/null

# Selenium
grep -l "selenium" requirements.txt pyproject.toml 2>/dev/null

Step 3: Discover Dev Server Commands

# Node.js
grep -E '"(dev|start|serve)"' package.json 2>/dev/null

# Makefile
grep -E "^(dev|serve|run|start|up):" Makefile 2>/dev/null

# Docker
ls docker-compose.yml docker-compose.yaml compose.yml compose.yaml 2>/dev/null

# Python
ls manage.py 2>/dev/null && echo "django: python manage.py runserver"
grep -E "(uvicorn|gunicorn|flask)" pyproject.toml requirements.txt 2>/dev/null

# Go
ls main.go cmd/*/main.go 2>/dev/null && echo "go: go run ."

# Monorepo
ls turbo.json 2>/dev/null && echo "turbo: check turbo.json for dev pipeline"

Step 4: Discover Port Configuration

# Check for port configuration
grep -rE "PORT|:3000|:8080|:5173|:4000|:8000|:3001" package.json .env .env.local .env.development 2>/dev/null | head -10

# Common framework defaults:
# Vite/SvelteKit=5173, Next.js/Rails=3000, CRA=3000
# Django=8000, Flask=5000, Go=8080, Spring=8080

Step 5: Check for Health/Readiness Endpoints

grep -rn "health\|healthz\|ready\|alive\|ping" --include="*.ts" --include="*.py" --include="*.go" --include="*.java" . 2>/dev/null | head -10

Runtime Verification Protocol

Step 1: Service Startup (if applicable)

If a dev server command is discovered:

1. Start server in background with PID tracking:

   {dev_cmd} &
   DEV_PID=$!
   

2. Wait for ready signal — try common health paths, fall back to port check:

   PORT={detected_port:-3000}
   for i in $(seq 1 $DEV_STARTUP_TIMEOUT); do
     curl -sf http://localhost:$PORT/ > /dev/null 2>&1 && break
     curl -sf http://localhost:$PORT/health > /dev/null 2>&1 && break
     curl -sf http://localhost:$PORT/healthz > /dev/null 2>&1 && break
     curl -sf http://localhost:$PORT/api/health > /dev/null 2>&1 && break
     nc -z localhost $PORT 2>/dev/null && break
     sleep 1
   done
   

3. If server doesn't start within ${DEV_STARTUP_TIMEOUT}s (default: 30s, configurable via .timeouts.devServerStartup), report as verification failure with the last few lines of output

Step 2: Smoke Tests

For each new/modified API endpoint in the diff:

Detecting endpoints from the diff:

# Find route definitions in changed files
git diff origin/$DEFAULT_BRANCH..HEAD --name-only | xargs grep -nE \
  "@(app|router)\.(get|post|put|patch|delete)|app\.(get|post|put|use)|router\.(get|post)|@(Get|Post|Put|Delete|Patch)Mapping|@api_view|path\(" \
  2>/dev/null

For each discovered endpoint:

1. Send a basic request and verify non-error response
2. Verify response structure matches expected schema
3. Test with invalid input and verify error handling

Step 3: E2E Tests (if framework detected)

Run discovered E2E test command with a timeout to prevent hanging:

timeout $E2E_TIMEOUT npx playwright test 2>&1  # or
timeout $E2E_TIMEOUT npx cypress run 2>&1       # or
timeout $E2E_TIMEOUT pytest tests/e2e/ 2>&1     # etc.

If the full suite is too large, run only tests related to changed files:

# Playwright: run specific test file
timeout $E2E_TIMEOUT npx playwright test tests/e2e/changed-feature.spec.ts 2>&1

# Pytest: run tests matching changed module names
timeout $E2E_TIMEOUT pytest tests/e2e/ -k "changed_module" 2>&1

Step 4: Visual Verification (if UI changes detected)

When the diff includes frontend changes (templates, components, styles, layouts), visually verify the running application in a browser. This catches rendering issues, broken layouts, and visual regressions that automated tests often miss.

Detecting UI changes in the diff:

git diff origin/$DEFAULT_BRANCH..HEAD --name-only | grep -iE "\.(tsx|jsx|vue|svelte|html|css|scss|sass|less|ejs|hbs|pug)$"

If UI files changed and the dev server is running:

Option A: Browser tool (preferred)

If a browser MCP tool is available (e.g., Puppeteer, Playwright MCP), use it to:

1. Navigate to the affected pages/routes
2. Take screenshots for evidence
3. Check for console errors
4. Verify interactive elements respond to clicks

Navigate to http://localhost:{PORT}{route}
Take a screenshot
Check the browser console for errors

Option B: Playwright screenshot script

If no browser MCP but Playwright is installed, take automated screenshots:

# Quick screenshot of affected pages
npx playwright test --project=chromium -g "screenshot" 2>&1 || \
npx playwright screenshot http://localhost:$PORT{route} screenshot-{route-name}.png 2>&1

Option C: WebFetch + curl fallback

If no browser automation is available, fetch the page HTML and verify structure:

# Fetch rendered HTML and check for expected elements
curl -s http://localhost:$PORT{route} | grep -E "<(main|section|div|h1)" | head -20

Use the WebFetch tool for richer inspection — it renders JavaScript and returns the page content, which is useful for SPAs where curl only sees a shell <div id="root">.

What to verify visually

Check	How	Evidence
Page loads without errors	Browser console or curl status	Screenshot or 200 OK
Layout isn't broken	Screenshot or HTML structure check	Screenshot file path
New UI elements are present	Look for expected elements in DOM	Element found / not found
No console errors or warnings	Browser console output	Clean console or error list
Interactive elements work	Click/interact via browser tool	Before/after screenshots

Record results in the Visual Verification section of the output.

Step 5: Acceptance Criteria Verification

For each acceptance criterion from the linked issue:

1. Identify how to verify it (API call, UI check, CLI command)
2. Execute the verification
3. Record pass/fail with evidence

Step 6: Cleanup

Kill any background services started in Step 1:

kill $DEV_PID 2>/dev/null
# For Docker Compose
docker compose down 2>/dev/null

Output Format

## Runtime Verification Results

### Service Status
| Service | Command | Status | Notes |
|---------|---------|--------|-------|
| Dev server | npm run dev | Started on :3000 | Healthy after 3s |

### Smoke Tests
| Endpoint/Feature | Test | Status | Evidence |
|-----------------|------|--------|----------|
| POST /api/users | Create user with valid data | Pass | 201 Created |
| POST /api/users | Create user with invalid email | Pass | 400 Bad Request |

### E2E Tests
| Suite | Status | Passed | Failed |
|-------|--------|--------|--------|
| Playwright | Pass | 12 | 0 |

### Visual Verification
| Page/Route | Check | Status | Evidence |
|------------|-------|--------|----------|
| /dashboard | Page loads | Pass | Screenshot: screenshot-dashboard.png |
| /dashboard | No console errors | Pass | Clean console |
| /users/new | Form renders correctly | Pass | All fields present |

### Acceptance Criteria
| Criterion | Verification Method | Status | Evidence |
|-----------|-------------------|--------|----------|
| Users can filter by date | GET /api/users?date=2024-01-01 | Pass | Returns filtered results |

### Not Verified (Requires Manual Check)
| Item | Reason |
|------|--------|
| Visual styling matches mockup | No browser automation available |

Graceful Degradation

Missing Capability	Fallback
No dev server command	Skip service startup, run only static checks
No E2E framework	Skip E2E, note as unverified
No health endpoint	Poll port availability with nc -z instead
No verification commands in CLAUDE.md	Infer from tech stack, ask user if ambiguous
Server won't start	Report failure with logs, don't block workflow
E2E tests timeout	Report timeout, suggest running a subset
No browser tool	Use Playwright screenshots, then WebFetch, then curl HTML check
No UI changes in diff	Skip visual verification entirely

Integration Points

This skill is invoked by:

• gh-start — Phase 7 (after quality checks, before code review)
• gh-pr — Step 3.6 (pre-PR runtime verification)

IMPORTANT: Runtime verification is additive, not blocking. If a project has no dev server or E2E framework, this skill completes with "skipped" status and the workflow continues.