validation-strategies

Memory

Search first: /mnemonic:search {relevant_keywords} Capture after: /mnemonic:capture {namespace} "{title}"

Run /mnemonic:list --namespaces to see available namespaces from loaded ontologies.

Validation Strategies

Choose and implement effective validation strategies for hook-driven tests.

Mnemonic Integration

Before advising on validation, check mnemonic for insights:

# Search for validation-related memories
rg -i "validation|assertion|regex|contains|flaky" ~/.claude/mnemonic/ ./.claude/mnemonic/ --glob "*.memory.md" -l | head -5

# Check learnings for validation experiences
rg -l "." ~/.claude/mnemonic/*/learnings/ --glob "*.memory.md" 2>/dev/null | xargs grep -l -i "validation\|expect" 2>/dev/null | head -5

Apply recalled insights:

• Validation approaches that worked well
• Flaky test patterns to avoid
• Regex patterns that proved reliable
• User's preferred validation strategies

Validation Approaches

String Matching (contains)

Strengths:

• Simple and readable
• Tolerant of formatting changes
• Fast execution

Weaknesses:

• May match unintended content
• No structure awareness
• Case-sensitive by default

Best for:

• Success/error messages
• UI text validation
• Presence checks

expect:
  - contains: "Operation successful"
  - contains: "Created"

Negative Matching (not_contains)

Strengths:

• Clear failure detection
• Complements positive checks
• Security validation

Weaknesses:

• Can't validate what IS present
• May miss variant error messages

Best for:

• Error absence verification
• Security checks (no sensitive data)
• Deprecation validation

expect:
  - not_contains: "error"
  - not_contains: "exception"
  - not_contains: "password"

Pattern Matching (regex)

Strengths:

• Precise structure validation
• Value extraction capability
• Multiple value matching

Weaknesses:

• Complex to write correctly
• Harder to debug failures
• Performance overhead

Best for:

• ID/UUID validation
• Numeric output
• Structured data
• Variable capture

expect:
  - regex: "ID:\\s+[a-f0-9]{12}"
  - regex: "Count:\\s+\\d+"
  - regex: "(success|complete|done)"

Choosing the Right Strategy

Decision Matrix

Scenario	Recommended	Why
Check operation succeeded	contains	Flexible message matching
Verify no errors	not_contains	Catches error variants
Extract ID for later use	regex with capture	Structured extraction
Validate numeric output	regex	Pattern precision
Check multiple valid responses	regex with alternation	(opt1|opt2)
Verify exact format	regex	Full control

Combination Strategies

Use multiple expectations for robust validation:

expect:
  # Positive confirmation
  - contains: "Memory created"
  # Negative confirmation
  - not_contains: "error"
  - not_contains: "failed"
  # Structured validation
  - regex: "ID:\\s+([a-f0-9]+)"

Regex Patterns Reference

Common Patterns

Pattern	Matches	Use Case
\\d+	One or more digits	Counts, IDs
[a-f0-9]+	Hex string	UUIDs, hashes
\\w+	Word characters	Identifiers
.*	Anything (greedy)	Flexible matching
.*?	Anything (non-greedy)	Minimal matching
\\s+	Whitespace	Flexible spacing
(a|b|c)	Alternation	Multiple valid values

ID Patterns

# UUID v4
regex: "[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}"

# Short ID (12 hex chars)
regex: "[a-f0-9]{12}"

# Numeric ID
regex: "ID:\\s+(\\d+)"

# Alphanumeric ID
regex: "ID:\\s+([a-zA-Z0-9]+)"

Output Patterns

# Success with count
regex: "Found\\s+(\\d+)\\s+results"

# Status messages
regex: "Status:\\s+(active|inactive|pending)"

# Version numbers
regex: "v(\\d+\\.\\d+\\.\\d+)"

# Timestamps
regex: "\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}"

Handling Edge Cases

Empty Results

Validate empty states explicitly:

- id: search_no_results
  action: "Search for 'nonexistent_term_xyz'"
  expect:
    - regex: "(no results|0 found|empty)"
    - not_contains: "error"

Variable Output

Handle outputs that may vary:

expect:
  # Accept range of counts
  - regex: "Found\\s+[1-9]\\d*\\s+results"
  # Accept multiple success messages
  - regex: "(created|updated|saved)"

Whitespace Sensitivity

Use \s+ for flexible whitespace:

# Rigid (may fail)
- contains: "ID: abc123"

# Flexible (handles formatting)
- regex: "ID:\\s+abc123"

Case Sensitivity

Regex can handle case variations:

# Case-insensitive with (?i)
- regex: "(?i)success"

# Explicit alternatives
- regex: "(Success|SUCCESS|success)"

Handling Flaky Tests

Causes of Flakiness

1. Timing issues: Async operations not complete
2. Order dependence: Tests affect each other
3. External dependencies: Network, services
4. Non-deterministic output: Timestamps, random IDs

Mitigation Strategies

1. Flexible expectations:

expect:
  # Don't match exact timestamp
  - regex: "Created at: \\d{4}-\\d{2}-\\d{2}"
  # Don't match exact ID
  - regex: "ID:\\s+[a-f0-9]+"

2. Retry logic (in runner):

cmd_validate_with_retry() {
  local max_retries=3
  for i in $(seq 1 $max_retries); do
    if cmd_validate "$1"; then
      return 0
    fi
    sleep 1
  done
  return 1
}

3. Tag flaky tests:

- id: external_api_test
  tags: [flaky, external]

4. Isolate stateful tests:

- id: cleanup_before_test
  action: "Reset test state"
  tags: [setup]

- id: actual_test
  depends_on: cleanup_before_test

Debugging Validation Failures

Reading Failure Messages

❌ FAIL: search_basic
Failures:
  - Missing: 'results found'
  - Pattern not found: 'Count:\s+\d+'

Interpretation:

• Missing: 'X' → contains: X failed
• Unexpected: 'X' → not_contains: X failed
• Pattern not found: 'X' → regex: X failed

Troubleshooting Steps

1. Get actual response: Check what Claude returned
2. Compare expectations: Look for typos, case issues
3. Test regex separately: Use online regex tester
4. Simplify expectations: Start broad, narrow down
5. Check for invisible chars: Whitespace, newlines

Common Fixes

Symptom	Likely Cause	Fix
Contains fails on visible text	Case mismatch	Use regex with (?i)
Regex fails on structured output	Escaping issue	Double-escape special chars
Intermittent failures	Timing/async	Add flexible patterns
All tests fail	State corruption	Reset with /run-tests --reset

Additional Resources

Reference Files

• references/validation-approaches.md - Validation strategies and patterns