shipwright-test

Shipwright Test Skill

Profile-aware test execution across all test layers.

How invoked: directly via /shipwright-test [--fix|--e2e-only|--design-fidelity|--report-boundary-coverage], or by /shipwright-run (orchestrator).

Step Index — Where the prose lives

Section	Reference
First Actions (A. Banner / B. Detect Profile / B2. Mode / B3. Context / B4. Prereqs / C. Strategy)	first-actions · prerequisite-self-healing
Step 0: Phase Session Context Recovery	step-0-phase-session
Step 1: Run Unit Tests	step-1-unit-tests
Step 1.5: Run Integration Tests	step-1.5-integration-tests
Step 1.6: Run pgTAP Database Tests	step-1.6-pgtap-tests
Step 2: Run Smoke Test	step-2-smoke-test
Step 2.5: Generate E2E Specs from Plan	step-2.5-e2e-spec-generation
Step 3: Run Playwright E2E	step-3-playwright-e2e
Step 3.5: E2E Results Verification	step-3.5-e2e-verification
Step 3.6: Cross-Page UI Consistency Check	step-3.6-ui-consistency
Step 3.7: Design Fidelity Verification	step-3.7-design-fidelity
Step 3.8: Performance Budget Check	step-3.8-performance-budget
Step 3.9: Stop Dev Server (always — finally-block)	step-3.9-stop-dev-server
Step 3.95: Boundary Coverage Report	step-3.95-boundary-coverage
Step 4: Security Scan -> /shipwright-security	step-4-security
Step 5: Report Results	step-5-report-results · results-enforcement · completion-gate
Anti-Rationalization	anti-rationalization
Test layers strategy (legacy)	test-layers
Reflection	reflection

---

CRITICAL: First Actions

Governing rules: Read and follow shared/constitution.md (ALWAYS / ASK FIRST / NEVER boundaries).

A. Print Intro Banner

================================================================================
SHIPWRIGHT-TEST: Test Runner
================================================================================
Runs tests across all layers based on stack profile.

Usage: /shipwright-test
   or: /shipwright-test --fix        (auto-fix failures, max 3 retries)
   or: /shipwright-test --e2e-only   (only Playwright E2E)
   or: /shipwright-test --design-fidelity  (only design fidelity check)
   or: /shipwright-test --report-boundary-coverage  (audit Affected Boundaries declarations across iterate-specs, ADR-027)
   or: Invoked by /shipwright-run (orchestrator)

Test layers:
  1.  Unit tests (Vitest / pytest)
  1.5 Integration tests (if profile has testing.integration)
  1.6 pgTAP database tests (if supabase/tests/database/ exists)
  2.  Smoke test (HTTP 200 on DEV URL)
  3.  Playwright E2E (if UI project + DEV URL available)
  3.6 Cross-page consistency (if .shipwright/designs/visual-guidelines.md exists)
  3.7 Design fidelity (if .shipwright/designs/screen-routes.json exists)
  3.8 Performance budget (if profile has testing.performance.enabled)
  3.9 Stop dev server (always, finally-block — runs even if 3.7 / 3.8 failed)
  4.  Security scan → out-of-band; see /shipwright-security (not auto-invoked)
================================================================================

B. Detect Profile

Read shipwright_project_config.json from project root and load profile from {plugin_root}/../../shared/profiles/{profile}.json. If no config, detect from package.json / pyproject.toml. Full detection rules: see first-actions.

B2. Detect Invocation Mode

Resolve it with {shared_root}/scripts/tools/get_phase_context.py --phase-task-id "{phaseTaskId}" --phase test (omit --phase-task-id entirely if the orchestrator did not hand you one — that is what selects standalone) and store the returned mode as invocation_mode. The dispatch token the orchestrator hands you is the authority — never re-derive the mode from run-config state. pipeline → full integration. standalone (no token) → add "mode": "standalone" to shipwright_test_results.json, skip pipeline state updates, and ASK first when requires_out_of_sequence_warning is true. error → STOP (never continue as standalone: the validator rejects standalone-stamped results and the run deadlocks). Full rules: see first-actions.

B3. Load Project Context

Read .shipwright/agent_docs/architecture.md and previous shipwright_test_results.json. Missing files: skip silently.

B4. Prerequisite Self-Healing

Auto-generate missing dev_url, .shipwright/designs/visual-guidelines.md, .shipwright/designs/screen-routes.json, .shipwright/planning/claude-plan-e2e.md, and playwright.config.ts where source data exists. Full rules and per-artifact commit hints: see prerequisite-self-healing.

C. Determine Test Strategy

Based on profile:

Profile	Unit Runner	E2E	Smoke URL Pattern
supabase-nextjs	npx vitest run	Playwright	http://localhost:3000
(future)	configurable	configurable	configurable

---

Step 0: Phase Session Context Recovery

If the orchestrator handed you a phaseTaskId, run get_phase_context.py --phase-task-id <id> as the first action. Full body: see step-0-phase-session.

Step 1: Run Unit Tests

uv run "{plugin_root}/scripts/lib/test_runner.py" --profile "{profile}" --layer unit

Autonomous mode auto-applies --fix behavior. Max 3 retries, structured debugging (root cause -> pattern check -> hypothesis -> fix -> re-run). Full body: see step-1-unit-tests.

Step 1.5: Run Integration Tests

Skip if no testing.integration profile config OR tests/integration/ missing. CI: missing env vars = FAIL; locally: skip with warning. Fast-fail on ECONNREFUSED/ETIMEDOUT and on >50% simultaneous failure. Never auto-fix by weakening RLS, swapping to service-role, or disabling URL safety. Full body: see step-1.5-integration-tests.

Step 1.6: Run pgTAP Database Tests

Skip if supabase/tests/database/ missing. Run via supabase test db or the shared test_runner.py --layer pgtap. Autofix as for integration. Full body: see step-1.6-pgtap-tests.

Step 2: Run Smoke Test (if DEV URL available)

uv run "{shared_root}/scripts/smoke_test.py" --url "{dev_url}" --timeout 10 --health-path "/api/health"

DEV URL sources (in order): shipwright_build_config.json.dev_url -> SHIPWRIGHT_DEV_URL env -> http://localhost:3000 default. The smoke-test script is shared ({shared_root}/scripts/smoke_test.py), not a project file. On failure: diagnose, autonomous fix, retry, then ASK after 2. Full body: see step-2-smoke-test.

Step 2.5: Generate E2E Specs from Plan (if missing)

Skip if e2e/ already has .spec.ts files, or profile has no UI. Otherwise scan .shipwright/planning/ for claude-plan-e2e.md and generate e2e/flows/*.spec.ts, e2e/pages/*.page.ts, e2e/fixtures/*, and e2e/fixtures/auth.setup.ts. Full body and structure: see step-2.5-e2e-spec-generation.

Step 3: Run Playwright E2E (if applicable)

Prereqs: UI profile + smoke test passed. Run playwright_setup.py, dev_server.py start, then playwright_runner.py --cwd {project_root}. Auto-fix groups failures by root cause (auth / selector / data / network / other), fixes per group with commits, parks unresolvable groups after 3 attempts and ASKs the user. Full body: see step-3-playwright-e2e.

Step 3.5: E2E Results Verification

Read e2e-results.json (Playwright authoritative). Filter to the chromium project (exclude setup). Reconcile shipwright_test_results.json.e2e with Playwright's expected + unexpected + skipped, write an e2e_verification_note if divergent, and verify playwright-report/index.html. Full body: see step-3.5-e2e-verification.

Step 3.6: Cross-Page UI Consistency Check (if applicable)

Runs if .shipwright/designs/visual-guidelines.md exists AND profile has UI (component_library set), or standalone via --consistency. Non-blocking (WARNING). Outliers are grouped by Spacing / Components / Colors root cause and fixed with per-group commits, max 3 retries per category. Full body: see step-3.6-ui-consistency.

Step 3.7: Design Fidelity Verification — Regressions-Only (if applicable)

Runs if .shipwright/designs/screen-routes.json exists, or standalone via --design-fidelity. Non-blocking (WARNING). Triages needs_review screens against design-fidelity-report.json into Resolved / Regression / Persistent Failure / Unchecked. Deep-reviews flagged screens against 5 dimensions, max 3 retries each. Full body: see step-3.7-design-fidelity.

Step 3.8: Performance Budget Check (if applicable)

Runs if profile.testing.performance.enabled: true OR shipwright_test_config.json.performance.enabled: true. Lighthouse via Playwright Chromium; bundle measurement = gzip-compressed *.js/*.css in build_output_dir. Gate warn (default) vs block (opt-in). Override precedence: --gate -> shipwright_test_config.json.performance -> profile -> defaults (LH 85, bundle 250 KB gz, LCP 2500 ms, gate warn). Full body: see step-3.8-performance-budget.

Step 3.9: Stop Dev Server (always — finally-block semantics)

Run unconditionally as a cleanup pass — even if Step 3.7 or Step 3.8 raised or failed. A blocked test phase is recoverable; a zombie dev server bound to the dev port is not.

uv run "{plugin_root}/../../shared/scripts/dev_server.py" stop --cwd {project_root}

Full body: see step-3.9-stop-dev-server.

Step 3.95: Boundary Coverage Report (if --report-boundary-coverage)

Runs only when invoked with --report-boundary-coverage OR shipwright_test_config.json.boundary_coverage.enabled: true. Audit hook for ADR-024. Scans .shipwright/planning/iterate/**/*.md for ## Affected Boundaries, correlates against shipwright_events.jsonl, writes coverage summary + drift signals. Non-blocking (WARNINGs in test summary). Full body: see step-3.95-boundary-coverage.

Step 4: Security Scan → /shipwright-security

Security scanning runs out-of-band; /shipwright-run does NOT auto-invoke it (decoupled in iterate sec-report-and-orchestrator-decouple, 2026-04). This step is a no-op in shipwright-test. Full body: see step-4-security.

Step 5: Report Results

Print the SHIPWRIGHT-TEST RESULTS summary (unit / integration / pgTAP / smoke / e2e / consistency / design fidelity / performance / security / overall) and the preview banner when the profile has UI. Capture learnings into conventions.md#Learnings. Record test_run + phase_completed events, update build dashboard, write canon-marker handoff, append phase_history, then mark the test step complete via the orchestrator.

Per-layer enforcement: see results-enforcement. Completion gate (every layer must have an explicit result): see completion-gate. Full body: see step-5-report-results.

---

Anti-Rationalization

See anti-rationalization — the table of common rationalizations ("Tests pass = correct", "E2E flaky = ignore", "We'll add more later", etc.) and the reality check for each.