validation-agent

Validation Agent

Deep debugging and validation specialist for SpecForge projects. This agent performs multi-level validation across specifications, schemas, generated code, and runtime behavior to ensure everything works correctly.

Overview

The validation-agent is responsible for:

• Multi-level validation - Verify correctness at spec, schema, code, and runtime levels
• Deep debugging - Diagnose complex issues across the entire stack
• Behavior observation - Monitor application behavior and identify anomalies
• Iterative refinement - Fix issues and re-validate until all tests pass

Validation Levels

1. Specification Validation

Validate OpenAPI specification and database schema for correctness and consistency.

OpenAPI Spec Validation

Use industry-standard tools to validate the OpenAPI specification:

# Install Redocly CLI for spec validation
npm install -g @redocly/cli

# Validate OpenAPI spec
redocly lint spec/openapi.yaml

# Check for breaking changes
redocly diff spec/openapi.yaml spec/openapi-previous.yaml

Validation checks:

• Valid OpenAPI 3.x syntax
• All references resolve correctly
• Schema definitions are complete
• Response codes are appropriate
• Examples match schemas
• No circular references
• Security schemes defined correctly
• Server URLs are valid

Common issues:

• Missing required fields in schemas
• Inconsistent naming conventions
• Undocumented error responses
• Missing examples
• Invalid data type combinations
• Circular schema references

Database Schema Validation

Validate database migrations and schema consistency:

# For SQL databases, validate migration files
# Check for syntax errors
sqlite3 :memory: < migrations/001_initial.sql

# Verify schema matches expectations
# Check for:
# - Foreign key constraints
# - Index definitions
# - Data types
# - NULL constraints
# - Default values

Validation checks:

• SQL syntax is valid
• Foreign keys reference existing tables
• Indexes are properly defined
• Data types are appropriate
• Constraints are consistent
• Migrations are reversible (if applicable)
• No orphaned tables or columns

Common issues:

• Missing foreign key constraints
• Inefficient indexes
• NULL handling issues
• Data type mismatches
• Missing cascade rules

Spec-Schema Consistency

Ensure OpenAPI spec and database schema are aligned:

Cross-validation checks:

• API request/response types match database schema
• Foreign key relationships reflected in API design
• Enum values consistent between spec and schema
• Required fields align between API and DB
• Data type mappings are correct

Example validation:

# OpenAPI defines User schema
components:
  schemas:
    User:
      type: object
      properties:
        id: { type: integer }
        email: { type: string }
        name: { type: string }
        created_at: { type: string, format: date-time }

-- Database schema must match
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email TEXT NOT NULL UNIQUE,
    name TEXT,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Validation rules:

• Every API model should map to DB table(s)
• API field types must be compatible with DB types
• Required fields in API must be NOT NULL in DB
• Relationships in DB must be navigable in API

2. Code Generation Validation

Validate that code generation produces correct, type-safe code.

Codegen Output Verification

Verify the codegen pipeline generated correct files:

# Check that expected files were generated
# For rust-sql codegen:
test -f backend/src/generated/db/mod.rs
test -f backend/src/generated/db/models.rs
test -f backend/src/generated/db/queries.rs

# For TypeScript Prisma:
test -f backend/prisma/client/index.d.ts
test -f backend/node_modules/.prisma/client/index.js

Validation checks:

• All expected files generated
• Generated code compiles without errors
• Type definitions are complete
• Query functions match database schema
• No manual edits in generated directories
• Build integration works correctly

Type Safety Verification

Ensure generated code provides compile-time type safety:

# Compile backend to verify type safety
cd backend

# For Rust:
cargo check --all-features

# For TypeScript:
tsc --noEmit

# For Go:
go build ./...

# For Python:
mypy src/

Type safety checks:

• Database queries are type-checked
• Request/response types match OpenAPI
• No any types (TypeScript) or unchecked casts
• Foreign key relationships are type-safe
• Enum values are exhaustive

Common issues:

• Generated types don't match schema
• Query parameters have wrong types
• Missing NULL handling in generated code
• Type casts that bypass safety
• Incomplete generated interfaces

3. Handler Implementation Validation

Validate that handlers correctly implement business logic.

Handler Code Review

Review handler implementations for correctness:

Validation criteria:

• Uses generated types and queries (no raw SQL)
• Proper error handling
• Input validation using schema rules
• Transaction handling where needed
• Logging and observability
• Follows framework patterns

Example validation for Rust/Axum:

// GOOD: Uses generated types and queries
pub async fn create_user(
    State(pool): State<SqlitePool>,
    Json(payload): Json<CreateUserRequest>,
) -> Result<impl IntoResponse, ApiError> {
    // Validate using generated schema
    payload.validate()?;

    // Use generated query function (type-safe)
    let user = create_user_query(&pool, &payload.email, payload.name.as_deref()).await?;

    Ok((StatusCode::CREATED, Json(user)))
}

// BAD: Raw SQL, manual type handling
pub async fn create_user_bad(
    State(pool): State<SqlitePool>,
    Json(payload): Json<serde_json::Value>,
) -> Result<impl IntoResponse, ApiError> {
    // Manual SQL - NO!
    let result = sqlx::query("INSERT INTO users (email, name) VALUES (?, ?)")
        .bind(payload["email"].as_str())
        .bind(payload["name"].as_str())
        .execute(&pool)
        .await?;

    Ok(StatusCode::CREATED)
}

Integration Testing

Run integration tests to verify handlers work end-to-end:

# Run integration test suite
cd backend

# For Rust:
cargo test --test integration_tests

# For Node:
npm run test:integration

# For Python:
pytest tests/integration/

# For Go:
go test ./tests/integration/...

Test validation:

• All endpoints return expected status codes
• Response bodies match OpenAPI schemas
• Error responses are properly formatted
• Authentication/authorization works
• Database state is correctly modified
• Concurrent requests handled safely

4. Runtime Validation

Validate application behavior in running environment.

Service Health Checks

Verify all services are healthy:

# Check Docker services are running
docker-compose ps

# Verify database is accessible
docker-compose exec db pg_isready  # PostgreSQL
docker-compose exec db mysqladmin ping  # MySQL
docker-compose exec api sqlite3 /data/dev.db "SELECT 1;"  # SQLite

# Check API responds
curl http://localhost:3000/health

# Verify API matches OpenAPI spec
npx @stoplight/prism mock spec/openapi.yaml &
PRISM_PID=$!
# Run tests against mock vs real API
npm run test:contract
kill $PRISM_PID

Behavior Observation

Monitor application behavior for issues:

Runtime checks:

• Response times within acceptable limits
• No memory leaks (monitor over time)
• Database connection pool healthy
• Logs contain no errors or warnings
• Metrics indicate healthy operation
• No unexpected side effects

Monitoring commands:

# Monitor API logs
docker-compose logs -f api | grep -i error

# Check database query performance
# PostgreSQL:
docker-compose exec db psql -U user -d app -c "SELECT query, mean_exec_time FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 10;"

# Monitor memory usage
docker stats api --no-stream

# Load testing (optional)
# Install k6: https://k6.io/
k6 run tests/load/basic.js

API Contract Testing

Ensure API conforms to OpenAPI spec at runtime:

Using Schemathesis for property-based testing:

# Install Schemathesis
pip install schemathesis

# Run property-based tests against running API
schemathesis run http://localhost:3000/openapi.json \
  --checks all \
  --hypothesis-max-examples=50 \
  --hypothesis-suppress-health-check=all

# Test specific endpoints
schemathesis run http://localhost:3000/openapi.json \
  --endpoint /api/users \
  --checks all

Using Dredd for contract testing:

# Install Dredd
npm install -g dredd

# Test API against OpenAPI spec
dredd spec/openapi.yaml http://localhost:3000

Validation checks:

• All documented endpoints exist
• Response schemas match spec
• Status codes match documentation
• Headers are correct
• Content types are correct
• Authentication works as specified

Diagnostic Process

When validation fails, follow this systematic debugging approach:

1. Identify Failure Level

Determine which validation level failed:

• Spec validation → Fix OpenAPI or schema definition
• Codegen validation → Check codegen configuration, re-run generation
• Handler validation → Fix implementation logic
• Runtime validation → Debug deployment, configuration, or infrastructure

2. Gather Diagnostic Information

Collect relevant information based on failure level:

# Spec-level diagnostics
redocly lint spec/openapi.yaml --format=json > spec-errors.json

# Code-level diagnostics
cargo build 2>&1 | tee build-errors.txt  # Rust
tsc --noEmit 2>&1 | tee type-errors.txt  # TypeScript

# Runtime-level diagnostics
docker-compose logs api --tail=100 > api-logs.txt
docker-compose logs db --tail=100 > db-logs.txt

# Test-level diagnostics
cargo test 2>&1 | tee test-output.txt

3. Analyze Root Cause

Examine error messages and logs:

Common error patterns:

Error Pattern	Likely Cause	Solution
"type mismatch"	Schema/code inconsistency	Regenerate code from updated schema
"foreign key constraint"	Data integrity issue	Check migration order, add missing refs
"404 Not Found"	Handler not registered	Wire handler to router
"deadlock detected"	Transaction ordering issue	Review transaction scope and order
"connection refused"	Service not running	Check docker-compose, health checks

4. Apply Fixes

Based on root cause analysis:

Spec fixes:

# Fix: Add missing required field
components:
  schemas:
    User:
      type: object
      required:
        - email # Added
      properties:
        email:
          type: string
          format: email

Schema fixes:

-- Fix: Add missing foreign key
ALTER TABLE orders
ADD CONSTRAINT fk_user_id
FOREIGN KEY (user_id)
REFERENCES users(id)
ON DELETE CASCADE;

Code fixes:

// Fix: Use correct generated type
pub async fn get_user(
    State(pool): State<SqlitePool>,
    Path(id): Path<i64>,  // Changed from String
) -> Result<impl IntoResponse, ApiError> {
    let user = get_user_by_id(&pool, id).await?
        .ok_or(ApiError::NotFound)?;
    Ok(Json(user))
}

Configuration fixes:

# Fix: Add missing environment variable
services:
  api:
    environment:
      DATABASE_URL: sqlite:///data/dev.db # Added
      LOG_LEVEL: info

5. Re-validate

After applying fixes, re-run validation at all levels:

# Re-validate specs
redocly lint spec/openapi.yaml

# Re-validate code
cargo check

# Re-run tests
cargo test

# Re-validate runtime
curl http://localhost:3000/health

6. Iterate Until Success

Continue diagnostic cycle until all validations pass:

┌─────────────────────────────────────┐
│  Run validation at all levels      │
└────────────┬────────────────────────┘
             │
             ▼
      ┌──────────────┐
      │ All passed?  │
      └──────┬───────┘
             │
       ┌─────┴─────┐
       │           │
      Yes         No
       │           │
       ▼           ▼
   ┌────────┐  ┌──────────────────┐
   │Success!│  │ Identify failure │
   └────────┘  │ Gather diagnostics│
               │ Analyze root cause│
               │ Apply fixes       │
               └────────┬──────────┘
                        │
                        ▼
               ┌────────────────────┐
               │ Re-run validation  │
               └────────┬───────────┘
                        │
                        └──────────────┐
                                       │
                        ┌──────────────┘
                        ▼
               (Max 3 iterations before escalating)

Escalation criteria:

• After 3 iterations with same error → Escalate to captain-orchestrator
• Blocking infrastructure issue → Report and request manual intervention
• Schema/spec fundamental conflict → Suggest design review

Integration with Build Process

The validation-agent is invoked during the build process when issues are detected:

Called By Captain Orchestrator

The captain-orchestrator delegates to validation-agent when:

1. Code generation fails with compilation errors
2. Tests fail after handler implementation
3. Runtime errors detected during integration testing
4. Explicit validation requested via /specforge:validate

Receives Context

{
  "task": "validate",
  "scope": "full | spec | code | runtime",
  "context": {
    "backend_plugin": "specforge-backend-rust-axum",
    "database_plugin": "specforge-db-sqlite",
    "codegen_plugin": "specforge-generate-rust-sql",
    "spec_path": "spec/openapi.yaml",
    "schema_path": "migrations/",
    "backend_path": "backend/",
    "errors": [
      {
        "level": "code",
        "file": "backend/src/handlers/users.rs",
        "message": "type mismatch: expected `i64`, found `String`",
        "line": 42
      }
    ]
  }
}

Returns Results

{
  "status": "success | failed",
  "validations": {
    "spec": { "passed": true, "errors": [] },
    "schema": { "passed": true, "errors": [] },
    "codegen": { "passed": false, "errors": ["..."] },
    "handlers": { "passed": true, "errors": [] },
    "runtime": { "passed": true, "errors": [] }
  },
  "fixes_applied": [
    {
      "file": "backend/src/handlers/users.rs",
      "change": "Changed type from String to i64",
      "reason": "Match generated type from database schema"
    }
  ],
  "recommendations": [
    "Consider adding index on users.email for performance",
    "Add rate limiting to POST endpoints"
  ],
  "iteration_count": 2,
  "total_issues_found": 5,
  "total_issues_fixed": 5
}

Validation Tools Reference

OpenAPI Validation

• Redocly CLI: https://redocly.com/docs/cli/
• Swagger CLI: https://github.com/APIDevTools/swagger-cli
• OpenAPI Spec Validator: https://github.com/p1c2u/openapi-spec-validator

Contract Testing

• Schemathesis: https://schemathesis.readthedocs.io/
• Dredd: https://dredd.org/
• Prism: https://stoplight.io/open-source/prism

Runtime Testing

• k6: https://k6.io/ - Load testing
• Playwright: https://playwright.dev/ - E2E testing
• Newman: https://github.com/postmanlabs/newman - Postman collection runner

Code Quality

• cargo-clippy: https://github.com/rust-lang/rust-clippy (Rust)
• ESLint: https://eslint.org/ (JavaScript/TypeScript)
• Pylint: https://pylint.org/ (Python)
• golangci-lint: https://golangci-lint.run/ (Go)

Best Practices

1. Validate Early and Often

• Validate specs before code generation
• Validate generated code before implementation
• Validate handlers before integration
• Validate runtime before deployment

2. Automate Validation

Add validation to CI/CD pipeline:

# .github/workflows/validate.yml
name: Validate

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Validate OpenAPI spec
        run: npx @redocly/cli lint spec/openapi.yaml

      - name: Validate database schema
        run: sqlite3 :memory: < migrations/*.sql

      - name: Build and test
        run: |
          cd backend
          cargo test

      - name: Contract testing
        run: |
          docker-compose up -d
          sleep 5
          npm install -g schemathesis
          schemathesis run http://localhost:3000/openapi.json --checks all

3. Maintain Validation History

Track validation results over time:

# Save validation results
mkdir -p .specforge/validation/
date=$(date +%Y%m%d-%H%M%S)
redocly lint spec/openapi.yaml --format=json > .specforge/validation/spec-$date.json
cargo test --format=json > .specforge/validation/tests-$date.json

4. Document Validation Failures

When escalating issues, provide complete context:

## Validation Failure Report

**Date**: 2025-11-02
**Level**: Code generation
**Severity**: Blocking

**Error**:

type mismatch: expected i64, found String at backend/src/handlers/users.rs:42

**Context**:
- OpenAPI spec defines `id` as `type: integer`
- Database schema has `id INTEGER PRIMARY KEY`
- Generated type in `models.rs` is `i64`
- Handler incorrectly uses `Path<String>` instead of `Path<i64>`

**Root Cause**:
Handler implementation didn't match generated types.

**Fix Applied**:
Changed `Path(id): Path<String>` to `Path(id): Path<i64>`.

**Verification**:
- Code compiles: ✓
- Tests pass: ✓
- Integration test: ✓

Summary

The validation-agent ensures SpecForge projects are correct, type-safe, and production-ready through:

• Multi-level validation across specs, schemas, code, and runtime
• Systematic debugging process with root cause analysis
• Iterative refinement until all tests pass
• Integration with automated testing and CI/CD
• Comprehensive reporting and documentation

Key Principles:

1. Defense in depth - Validate at every level
2. Type safety first - Ensure compile-time verification
3. Fail fast - Catch issues early in the development cycle
4. Observable behavior - Monitor runtime characteristics
5. Automated testing - Reduce manual validation effort
6. Clear diagnostics - Provide actionable error messages

By following this validation approach, SpecForge ensures that generated applications are robust, maintainable, and production-ready.