marsai:dev-sre

SRE Validation (Gate 2)

Overview

This skill VALIDATES that observability was correctly implemented by developers:

• Structured logging with trace correlation
• OpenTelemetry tracing instrumentation
• Code instrumentation coverage (90%+ required)
• Context propagation for distributed tracing

CRITICAL: Role Clarification

Developers IMPLEMENT observability. SRE VALIDATES it.

Who	Responsibility
Developers (Gate 0)	IMPLEMENT observability following MarsAI Standards
SRE Agent (Gate 2)	VALIDATE that observability is correctly implemented
Implementation Agent	FIX issues found by SRE (if any)

If observability is missing or incorrect:

1. SRE reports issues with severity levels
2. This skill dispatches fixes to the implementation agent
3. SRE re-validates after fixes
4. Max 3 iterations, then escalate to user

---

Step 1: Validate Input

<verify_before_proceed>

• unit_id exists
• language is valid (typescript|python)
• service_type is valid (api|worker|batch|cli|library)
• implementation_agent exists
• implementation_files is not empty </verify_before_proceed>

REQUIRED INPUT (from marsai:dev-cycle orchestrator):
- unit_id: [task/subtask being validated]
- language: [typescript|python]
- service_type: [api|worker|batch|cli|library]
- implementation_agent: [agent that did Gate 0]
- implementation_files: [list of files from Gate 0]

OPTIONAL INPUT:
- external_dependencies: [HTTP clients, gRPC clients, queues]
- gate0_handoff: [summary from Gate 0]
- gate1_handoff: [summary from Gate 1]

if any REQUIRED input is missing:
  → STOP and report: "Missing required input: [field]"
  → Return to orchestrator with error

Step 2: Initialize Validation State

validation_state = {
  iteration: 1,
  max_iterations: 3,
  sre_result: null,
  issues: [],
  instrumentation_coverage: null
}

Step 3: Dispatch SRE Agent for Validation

<dispatch_required agent="marsai:sre"> Validate observability implementation for unit_id. </dispatch_required>

Task:
  subagent_type: "marsai:sre"
  description: "Validate observability for [unit_id]"
  prompt: |
    ⛔ VALIDATE Observability Implementation

    ## Input Context
    - **Unit ID:** [unit_id]
    - **Language:** [language]
    - **Service Type:** [service_type]
    - **Implementation Agent:** [implementation_agent]
    - **Files to Validate:** [implementation_files]
    - **External Dependencies:** [external_dependencies or "None"]

    ## Standards Reference
    WebFetch: https://raw.githubusercontent.com/V4-Company/marsai/main/dev-team/docs/standards/sre.md

    ## Your Role
    - VALIDATE that observability is implemented correctly
    - Do not implement - only verify and report
    - Check structured JSON logging
    - Check OpenTelemetry instrumentation coverage
    - Check context propagation for external calls

    ## Validation Checklist

    ### 0. FORBIDDEN Logging Patterns (CRITICAL - Check FIRST)

    Any occurrence = CRITICAL severity, automatic FAIL verdict.

    <forbidden>
    - console.log() in TypeScript
    - console.error() in TypeScript
    - console.warn() in TypeScript
    </forbidden>
    
    **MUST search for and report all occurrences of FORBIDDEN patterns:**
    
    | Language | FORBIDDEN Pattern | Search For |
    |----------|-------------------|------------|
    | TypeScript | `console.log()` | `console.log` in *.ts files |
    | TypeScript | `console.error()` | `console.error` in *.ts files |
    | TypeScript | `console.warn()` | `console.warn` in *.ts files |
    
    **If any FORBIDDEN pattern found:**
    - Severity: **CRITICAL**
    - Verdict: **FAIL** (automatic, no exceptions)
    - Each occurrence MUST be listed with file:line
    
    ### 1. Structured Logging
    - [ ] Uses `initializeLogger()` from lib-common-js (TypeScript)
    - [ ] JSON format with timestamp, level, message, service
    - [ ] trace_id correlation in logs
    - [ ] **no FORBIDDEN patterns** (see check 0 above)

    ### 2. Instrumentation Coverage (90%+ required)
    For [language], check these patterns:

    **TypeScript:**
    ```typescript
    const span = tracer.startSpan('layer.operation');
    try { /* work */ } finally { span.end(); }
    ```

    Count spans in:
    - Handlers: grep "tracer.Start" in *controller*.ts
    - Services: grep "tracer.Start" in *service*.ts
    - Repositories: grep "tracer.Start" in *repository*.ts

    ### 3. Context Propagation
    For external calls, verify:
    - HTTP: context propagation middleware or equivalent
    - gRPC: context propagation interceptor or equivalent
    - Queues: header propagation or equivalent

    ### 4. Multi-Tenant Observability (MANDATORY)
    All services MUST include tenant context in observability:
    - [ ] Trace spans include `tenant_id` attribute when in multi-tenant mode
    - [ ] Structured logs include `tenant_id` field when in multi-tenant mode
    - [ ] Metrics include `tenant_id` label when in multi-tenant mode
    - [ ] Graceful degradation: no crash when `tenant_id` is absent (single-tenant mode)

    ## Required Output Format

    ### Validation Summary
    | Check | Status | Evidence |
    |-------|--------|----------|
    | Structured Logging | ✅/❌ | [file:line or "not FOUND"] |
    | Tracing Enabled | ✅/❌ | [file:line or "not FOUND"] |
    | Instrumentation ≥90% | ✅/❌ | [X%] |
    | Context Propagation | ✅/❌/N/A | [file:line or "N/A"] |
    | Multi-Tenant Observability | ✅/❌/N/A | [file:line where tenant_id in spans/logs/metrics, or "N/A" if single-tenant only] |

    ### Instrumentation Coverage Table
    | Layer | Instrumented | Total | Coverage |
    |-------|--------------|-------|----------|
    | Handlers | X | Y | Z% |
    | Services | X | Y | Z% |
    | Repositories | X | Y | Z% |
    | HTTP Clients | X | Y | Z% |
    | gRPC Clients | X | Y | Z% |
    | **TOTAL** | X | Y | **Z%** |

    ### Issues Found (if any)
    For each issue:
    - **Severity:** CRITICAL/HIGH/MEDIUM/LOW
    - **Category:** [Logging|Tracing|Instrumentation|Propagation]
    - **Description:** [what's wrong]
    - **File:** [path:line]
    - **Expected:** [what should exist]
    - **Fix Required By:** [implementation_agent]

    ### Verdict
    - **all CHECKS PASSED:** ✅ YES / ❌ no
    - **Instrumentation Coverage:** [X%]
    - **If no, blocking issues:** [list]

Step 4: Parse SRE Agent Output

Parse agent output:

1. Extract Validation Summary table
2. Extract Instrumentation Coverage table
3. Extract Issues Found list
4. Extract Verdict

validation_state.sre_result = {
  logging_ok: [true/false],
  tracing_ok: [true/false],
  instrumentation_coverage: [percentage],
  context_propagation_ok: [true/false/na],
  issues: [list of issues],
  verdict: [PASS/FAIL]
}

Step 5: Handle Validation Result

if validation_state.sre_result.verdict == "PASS" 
   and validation_state.sre_result.instrumentation_coverage >= 90:
  → Proceed to Step 8 (Success)

if validation_state.sre_result.verdict == "FAIL"
   or validation_state.sre_result.instrumentation_coverage < 90:
  → Proceed to Step 6 (Dispatch Fix)

if validation_state.iteration >= validation_state.max_iterations:
  → Proceed to Step 9 (Escalate)

Step 6: Dispatch Fix to Implementation Agent

Task:
  subagent_type: "[implementation_agent from input]"  # e.g., "marsai:backend-engineer-typescript"
  description: "Fix observability issues for [unit_id]"
  prompt: |
    ⛔ FIX REQUIRED - Observability Issues Found

    ## Context
    - **Unit ID:** [unit_id]
    - **Iteration:** [validation_state.iteration] of [validation_state.max_iterations]
    - **Your Previous Implementation:** [implementation_files]

    ## Issues to Fix (from SRE Validation)
    [paste issues from validation_state.sre_result.issues]

    ## Current Instrumentation Coverage
    [paste Instrumentation Coverage table from SRE output]
    **Required:** ≥90%
    **Current:** [validation_state.sre_result.instrumentation_coverage]%

    ## Standards Reference
    For TS: https://raw.githubusercontent.com/V4-Company/marsai/main/dev-team/docs/standards/typescript.md

    Focus on: Telemetry & Observability section

    ## Required Fixes

    ### If Logging Issues:
    - Replace fmt.Println/console.log with structured logger
    - Add trace_id to log context
    - Use JSON format

    ### If Instrumentation Coverage < 90%:
    - Add spans to all handlers: `tracer.Start(ctx, "handler.name")`
    - Add spans to all services: `tracer.Start(ctx, "service.domain.operation")`
    - Add spans to all repositories: `tracer.Start(ctx, "db.operation")`
    - Add `defer span.End()` after each span creation

    ### If Context Propagation Issues:
    - Add InjectHTTPContext for outgoing HTTP calls
    - Add InjectGRPCContext for outgoing gRPC calls
    - Add PrepareQueueHeaders for queue publishing

    ## Required Output
    - Files modified with fixes
    - New Instrumentation Coverage calculation
    - Confirmation all issues addressed

Step 7: Re-Validate After Fix

validation_state.iteration += 1

if validation_state.iteration > validation_state.max_iterations:
  → Proceed to Step 9 (Escalate)

→ Return to Step 3 (Dispatch SRE Agent)

Step 8: Success - Prepare Output

Generate skill output:

## Validation Result
**Status:** PASS
**Iterations:** [validation_state.iteration]
**Instrumentation Coverage:** [validation_state.sre_result.instrumentation_coverage]%

## Instrumentation Coverage
[paste final Instrumentation Coverage table]

## Issues Found
None (all resolved)

## Handoff to Next Gate
- SRE validation: COMPLETE
- Logging: ✅ Structured JSON with trace_id
- Tracing: ✅ OpenTelemetry instrumented
- Instrumentation: ✅ [X]% coverage
- Ready for Gate 3 (Testing): YES

Step 9: Escalate - Max Iterations Reached

Generate skill output:

## Validation Result
**Status:** FAIL
**Iterations:** [validation_state.iteration] (MAX REACHED)
**Instrumentation Coverage:** [validation_state.sre_result.instrumentation_coverage]%

## Instrumentation Coverage
[paste final Instrumentation Coverage table]

## Issues Found
[list remaining unresolved issues]

## Handoff to Next Gate
- SRE validation: FAILED
- Remaining issues: [count]
- Ready for Gate 3 (Testing): no
- **Action Required:** User must manually resolve remaining issues

⛔ ESCALATION: Max iterations (3) reached. User intervention required.

---

Severity Calibration

Severity	Scenario	Gate 2 Status	Action
CRITICAL	Missing all observability (no structured logs)	FAIL	❌ Return to Gate 0
CRITICAL	fmt.Println/echo instead of JSON logs	FAIL	❌ Return to Gate 0
CRITICAL	Instrumentation coverage < 50%	FAIL	❌ Return to Gate 0
CRITICAL	"DEFERRED" appears in validation output	FAIL	❌ Return to Gate 0
HIGH	Instrumentation coverage 50-89%	NEEDS_FIXES	⚠️ Fix and re-validate
MEDIUM	Missing context propagation	NEEDS_FIXES	⚠️ Fix and re-validate
LOW	Minor logging improvements	PASS	✅ Note for future

---

Blocker Criteria - STOP and Report

<block_condition> If any condition is true, STOP and dispatch fix or escalate to user.

• Service lacks JSON-structured logs
• Instrumentation coverage < 50%
• Max iterations (3) reached </block_condition>

Decision Type	Examples	Action
HARD BLOCK	Service lacks JSON structured logs	STOP - Dispatch fix to implementation agent
HARD BLOCK	Instrumentation coverage < 50%	STOP - Dispatch fix to implementation agent
HARD BLOCK	Max iterations reached	STOP - Escalate to user

---

Cannot Be Overridden

<cannot_skip>

• Gate 2 execution (no MVP exemptions)
• 90% instrumentation coverage minimum
• JSON structured logs requirement </cannot_skip>

Requirement	Cannot Be Waived By	Rationale
Gate 2 execution	CTO, PM, "MVP" arguments	Observability prevents production blindness
90% instrumentation coverage	"We'll add spans later"	Later = never. Instrument during implementation.
JSON structured logs	"Plain text is enough"	Plain text is unsearchable in production

Pressure Resistance

See shared-patterns/shared-pressure-resistance.md for universal pressure scenarios.

User Says	Your Response
"Skip SRE validation"	"Observability is MANDATORY. Dispatching SRE agent now."
"90% coverage is too high"	"90% is the MarsAI Standard minimum. Cannot lower."
"Will add instrumentation later"	"Instrumentation is part of implementation. Fix now."

---

Anti-Rationalization Table

See shared-patterns/shared-anti-rationalization.md for universal anti-rationalizations.

Gate 2-Specific Anti-Rationalizations

Rationalization	Why It's WRONG	Required Action
"OpenTelemetry library is installed"	Installation ≠ Instrumentation	Verify spans exist in code
"Middleware handles tracing"	Middleware = root span only	Add child spans in all layers
"Small function doesn't need span"	Size is irrelevant	Add span to every function
"Only external calls need tracing"	Internal ops need tracing too	Instrument all layers
"Feature complete, observability later"	Observability IS completion	Fix NOW before Gate 3

Component Type Requirements

Type	JSON Logs	Tracing	Instrumentation
API Service	REQUIRED	REQUIRED	90%+
Background Worker	REQUIRED	REQUIRED	90%+
CLI Tool	REQUIRED	N/A	N/A
Library	N/A	N/A	N/A

---

Execution Report Format

## Validation Result
**Status:** [PASS|FAIL|NEEDS_FIXES]
**Iterations:** [N]
**Duration:** [Xm Ys]

## Instrumentation Coverage
| Layer | Instrumented | Total | Coverage |
|-------|--------------|-------|----------|
| Handlers | X | Y | Z% |
| Services | X | Y | Z% |
| Repositories | X | Y | Z% |
| HTTP Clients | X | Y | Z% |
| gRPC Clients | X | Y | Z% |
| **TOTAL** | X | Y | **Z%** |

**Coverage Status:** [PASS (≥90%) | NEEDS_FIXES (50-89%) | FAIL (<50%)]

## Issues Found
- [List by severity or "None"]

## Handoff to Next Gate
- SRE validation status: [complete|needs_fixes|failed]
- Instrumentation coverage: [X%]
- Ready for testing: [YES|no]