cli-validation

CLI Validation

Scope

Command-line binary validation (Rust cargo, Go, Bun-compiled, Python scripts, Node CLI tools). Does NOT handle API servers (use api-validation) or GUI apps (use web-validation / ios-validation).

Preflight

• Toolchain present (cargo --version, go version, bun --version, etc.)
• build / cargo build --release / go build exits 0
• Produced binary at expected path AND executable (test -x)
• Required env vars present for runtime

Execution Pattern

# Capture stdout + stderr + exit code + timing in one pass
./target/release/mytool process --input ./fixtures/sample.csv --output ./out.json \
  2>e2e-evidence/vgN-stderr.log \
  > e2e-evidence/vgN-stdout.log
EXIT=$?
echo "exit=$EXIT" >> e2e-evidence/vgN-stdout.log

# Also capture the output file
cp ./out.json e2e-evidence/vgN-out.json 2>/dev/null || true

PASS Criteria Patterns

Good:

• "exit == 0"
• "stdout contains Processed 150 rows in"
• "stderr is empty or only contains INFO-level lines"
• "vgN-out.json has rows == 150 AND errors == 0"
• "file written at expected path with non-zero size"

Bad (reject):

• "Command runs" → exits 0? produces output? correct output?
• "No errors" → what about 2? exit code?

Gate Template

<validation_gate id="VG-N" blocking="true" platform="cli"
                 protocol="functional-validation"
                 platform_skill="cli-validation"
                 verdict_by="gate-validation-discipline">
  <preflight skill="preflight">[build clean + binary executable]</preflight>
  <execute>[./binary with real inputs; capture stdout/stderr/exit/artifacts]</execute>
  <capture>[all to e2e-evidence/vgN-*]</capture>
  <pass_criteria>[exit code + specific stdout strings + artifact checks]</pass_criteria>
  <review>[grep for markers; jq on output JSON; stat output files]</review>
  <verdict>PASS → next | FAIL → error-recovery → re-run</verdict>
  <mock_guard skill="no-mocking-validation-gates">
    No mocking stdin/stdout, no test-only flags, no dry-run for validation gates.
  </mock_guard>
</validation_gate>

Required Coverage

• Happy path: valid inputs → expected artifacts + exit 0
• Invalid args: → exit 2 (conventional) + helpful stderr, no partial writes
• Missing file: → exit 1 + clear error, no panic
• Help flag: → exit 0 + usage text
• Pipe in (if applicable): echo data | ./tool → correct behavior

Common Failure Patterns

Symptom	Likely cause	Fix
Exit 0 but wrong output	Silent error swallowed	Check return paths; propagate errors
Exit non-zero but no stderr	Panic before log init	Wire stderr logging first
Works on test inputs, crashes on real	Hardcoded assumption	Add input validation with clear error
Slow on large inputs	O(n²) in hot path	Profile before shipping

Rules

1. Always capture all three: stdout, stderr, exit code
2. Always read stdout — don't just check exit code
3. Always verify produced artifacts (file exists AND correct content)
4. Always cover error paths — exit-0-only = shipping bugs

Security Policy

CLIs may take secrets as args or env vars. Never log evidence containing real secrets; use placeholders when building fixture inputs.