ring:dev-readyz

Readyz & Self-Probe Implementation

Phase 1: Dependency Scan

Scan the project to detect ALL external dependencies:

# Go: detect imports and connection patterns
grep -rn 'pgx\|pgxpool\|mongo\.\|mongo-driver\|redis\.\|valkey\|amqp\|rabbitmq\|s3\|aws' go.mod internal/ pkg/ cmd/
grep -rn 'NewPostgres\|NewMongo\|NewRedis\|NewRabbit\|NewValkey\|WithModule' internal/

# TypeScript/Next.js: detect connection patterns
grep -rn 'MongoClient\|mongoose\|pg\|Pool\|redis\|amqplib\|S3Client' package.json src/ app/ lib/

Build dependency map: PostgreSQL (pgx), MongoDB (mongo-driver), Redis/Valkey (go-redis), RabbitMQ (amqp091-go), S3 (aws-sdk), HTTP clients. For each, detect if TLS is configured (sslmode, tls=true, rediss://, amqps://).

SaaS deployment mode: TLS is MANDATORY for all database connections. No exceptions.

Phase 2: /readyz Endpoint

Response Contract (MANDATORY)

{
  "status": "healthy",
  "checks": {
    "postgres": { "status": "up", "latency_ms": 2, "tls": true },
    "mongodb":  { "status": "up", "latency_ms": 3, "tls": true },
    "rabbitmq": { "status": "up", "connected": true },
    "valkey":   { "status": "up", "latency_ms": 1, "tls": false }
  },
  "version": "1.2.3",
  "deployment_mode": "saas"
}

• status: "healthy" if ALL checks pass, "unhealthy" if ANY fails
• Each check includes latency_ms (for connections with ping) and tls (boolean)
• deployment_mode: from DEPLOYMENT_MODE env or inferred from config
• version: from build info or VERSION env

Go Implementation (Fiber + lib-commons)

// internal/adapters/http/in/readyz.go

type DependencyCheck struct {
    Status    string `json:"status"`
    LatencyMs int64  `json:"latency_ms,omitempty"`
    TLS       *bool  `json:"tls,omitempty"`
    Connected *bool  `json:"connected,omitempty"`
    Error     string `json:"error,omitempty"`
}

type ReadyResponse struct {
    Status         string                      `json:"status"`
    Checks         map[string]DependencyCheck  `json:"checks"`
    Version        string                      `json:"version"`
    DeploymentMode string                      `json:"deployment_mode"`
}

func isCacheDependency(name string) bool {
    normalized := strings.ToLower(name)
    return strings.Contains(normalized, "redis") ||
        strings.Contains(normalized, "valkey") ||
        strings.Contains(normalized, "cache")
}

func ReadyHandler(deps Dependencies) fiber.Handler {
    return func(c *fiber.Ctx) error {
        ctx, cancel := context.WithTimeout(c.UserContext(), 5*time.Second)
        defer cancel()

        resp := ReadyResponse{
            Status:         "healthy",
            Checks:         make(map[string]DependencyCheck),
            Version:        buildVersion,
            DeploymentMode: os.Getenv("DEPLOYMENT_MODE"),
        }

        // Each check: ping + measure latency + verify TLS
        // Use 2s timeout per dependency, 1s for cache
        for name, checker := range deps.HealthCheckers() {
            timeout := 2 * time.Second
            if isCacheDependency(name) {
                timeout = 1 * time.Second
            }

            depCtx, depCancel := context.WithTimeout(ctx, timeout)
            check := checker.Check(depCtx)
            depCancel()

            resp.Checks[name] = check
            if check.Status != "up" {
                resp.Status = "unhealthy"
            }
        }

        if resp.Status != "healthy" {
            return libHTTP.ServiceUnavailable(c, "UNHEALTHY", "Service Unhealthy", resp)
        }
        return libHTTP.OK(c, resp)
    }
}

TLS Verification (CRITICAL)

Each checker MUST verify TLS state from the connection options (e.g., connOpts.TLSConfig != nil for Go, mongoClient.options?.tls for TS). This is what would have caught the Monetarie bug.

RabbitMQ note: The amqp091-go library's *amqp.Connection does not reliably expose TLS state after dialing. For RabbitMQ, TLS detection MUST inspect the connection URL scheme (amqps:// = TLS, amqp:// = plaintext). The checker constructor MUST accept the connection URL alongside the *amqp.Connection object and derive tls: true/false from the scheme. Do not attempt to reflect on the live connection object for this purpose.

SaaS TLS Enforcement

"SaaS deployment mode: TLS is MANDATORY" means two separate things that are both required:

Concern	Responsibility	Mechanism
Surface TLS state	/readyz probe	Reports "tls": true/false per dependency in JSON response
Enforce TLS	Bootstrap / connection code	MUST refuse to start if DEPLOYMENT_MODE=saas and TLS is not configured

MUST implement both. Surfacing without enforcement means the service starts silently insecure. Enforcement without surfacing means the Tenant Manager cannot confirm TLS posture post-provisioning. Neither alone is sufficient.

Bootstrap enforcement pattern (Go):

if os.Getenv("DEPLOYMENT_MODE") == "saas" && connOpts.TLSConfig == nil {
    return nil, fmt.Errorf("TLS is required in SaaS mode but not configured for %s", depName)
}

Next.js Implementation

Same pattern at app/api/admin/health/readyz/route.ts: ping each dependency, measure latency, check TLS, return 200/503 with the same JSON contract. Use Response.json() with appropriate status code.

Endpoint Paths

Stack	Ready Path	Health Path
Go API	/readyz	/health
Go Worker	/readyz on HEALTH_PORT	/health on HEALTH_PORT
Next.js	/api/admin/health/readyz	same as Ready Path

Next.js exposes a single /api/admin/health/readyz endpoint which serves both readiness and health checks.

Phase 3: Startup Self-Probe

The app MUST run all readiness checks at boot and log results BEFORE accepting traffic.

Go Implementation

// cmd/app/main.go or internal/bootstrap/selfprobe.go

func RunSelfProbe(ctx context.Context, deps Dependencies, logger Logger) error {
    logger.Infow("startup_self_probe_started",
        "probe", "self",
    )
    results := make(map[string]DependencyCheck)
    allHealthy := true

    for name, checker := range deps.HealthCheckers() {
        check := checker.Check(ctx)
        results[name] = check

        if check.Status == "up" {
            logger.Infow("self_probe_check",
                "probe", "self",
                "name", name,
                "status", check.Status,
                "duration_ms", check.LatencyMs,
                "tls", check.TLS,
            )
        } else {
            logger.Errorw("self_probe_check",
                "probe", "self",
                "name", name,
                "status", check.Status,
                "duration_ms", check.LatencyMs,
                "error", check.Error,
            )
            allHealthy = false
        }
    }

    if !allHealthy {
        logger.Errorw("startup_self_probe_failed",
            "probe", "self",
            "results", results,
        )
        return fmt.Errorf("self-probe failed: one or more dependencies unreachable")
    }

    logger.Infow("startup_self_probe_passed",
        "probe", "self",
        "results", results,
    )
    return nil
}

Impact on /health

Self-probe failure MUST affect /health:

var selfProbeOK atomic.Bool // package-level

func init() { selfProbeOK.Store(false) } // unhealthy until proven otherwise

// At startup, after self-probe succeeds:
if err := RunSelfProbe(ctx, deps, logger); err != nil {
    // selfProbeOK stays false — /health returns 503
    // K8s liveness probe will restart the pod
} else {
    selfProbeOK.Store(true)
}

// /health handler
f.Get("/health", func(c *fiber.Ctx) error {
    if !selfProbeOK.Load() {
        return libHTTP.ServiceUnavailable(c, "UNHEALTHY", "Self-probe failed", nil)
    }
    return libHTTP.HealthWithDependencies(deps)(c)
})

This is the key insight: /health is no longer just "process alive." It's "startup self-probe passed AND lib-commons runtime dependency state is healthy." A pod that starts but can't reach its databases will be restarted by K8s instead of silently serving errors, and runtime dependency or circuit-breaker failures are still surfaced through the standard lib-commons health handler.

Self-Probe Lifecycle

1. App starts → self-probe → logs each dep → /health reflects result
2. ALL pass: 200 on /health, /readyz operates normally
3. ANY fail: 503 on /health, K8s restarts pod via liveness probe
4. Optional: periodic re-probe via SELF_PROBE_INTERVAL env

Next.js Self-Probe Lifecycle

Next.js instrumentation.ts register() executes once at process startup and BLOCKS before the first request is served — this IS the self-probe point for Next.js. Use it.

MUST NOT call process.exit() on probe failure inside register(). Doing so prevents K8s from collecting a useful log tail. Instead:

1. In register(): run all dependency checks; if any fail, set a module-level flag (let startupHealthy = false).
2. The /api/admin/health/readyz route handler checks this flag.
3. Return 503 with the failed checks if the flag is false.
4. K8s readinessProbe hits /api/admin/health/readyz, sees 503, and withholds traffic — no process.exit() needed.

// instrumentation.ts
let startupHealthy = false;
let startupChecks: Record<string, DependencyCheck> = {};

export async function register() {
  const results = await runAllChecks();
  startupChecks = results;
  startupHealthy = Object.values(results).every(c => c.status === "up");
  // log results here — process stays alive regardless
}

export { startupHealthy, startupChecks };

The /api/admin/health/readyz route imports startupHealthy and startupChecks from instrumentation.ts and returns 200 or 503 accordingly.

Runtime vs Startup

These two mechanisms are complementary, not redundant:

Mechanism	When	Purpose
Self-probe	STARTUP — before first request	Validates dependencies are reachable before traffic is allowed
/readyz	RUNTIME — per request	Validates dependencies are still reachable as K8s readinessProbe
/health	RUNTIME — per request	Reflects self-probe result AND lib-commons runtime circuit-breaker state

A pod that passes startup self-probe can still fail /readyz later (e.g., DB goes away mid-run). A pod that fails self-probe should never receive traffic in the first place. Both gates are necessary.

Phase 4: Validation

Verify /readyz endpoint, RunSelfProbe function, and /health self-probe wiring all exist.

Checklist

• All detected dependencies have a checker in /readyz
• Each checker validates TLS when TLS is configured
• Each checker has a timeout (2s DB, 1s cache)
• Response includes per-dep latency and TLS status
• Startup self-probe runs before accepting traffic
• Self-probe results logged as structured JSON
• /health returns 503 if self-probe failed
• Helm values use /readyz for readinessProbe
• SaaS mode enforces TLS on all DB connections

Anti-Rationalization Table

Rationalization	Why It's WRONG	Required Action
"K8s TCP probe is enough"	TCP ≠ app ready. Monetarie incident: pod alive, Mongo dead.	Implement /readyz
"/health covers it"	/health without self-probe is blind to dep failures	Add self-probe, wire to /health
"TLS check is overhead"	TLS mismatch = silent failure for every query	Check TLS per dependency
"Only backend needs this"	Console (frontend) caused the incident	All apps, no exceptions
"Dependencies are reliable"	Networks partition. Configs drift. Certs expire.	Check every time
"Too many checks slow startup"	Bounded per-dependency timeouts keep overhead low. Incident costs hours.	No excuse
"Service has only one dependency"	One broken dependency = total outage. Complexity argument is irrelevant at zero scale. Self-probe is three lines of code.	Implement self-probe, no exceptions