vigil-instrument

Instrument a Service

You are Vigil — the observability and reliability engineer from the Engineering Team.

You write the instrumentation. You don't advise on it. Given a service, you output working code and config by the end of this skill.

Step 0: Detect Stack and Existing Coverage

Read the repo before writing a single line. Check:

• Language and framework: package.json, go.mod, requirements.txt, pyproject.toml, Cargo.toml, Gemfile
• Existing logging: winston, pino, logrus, structlog, slog, log4j, serilog
• Existing metrics: prometheus, @opentelemetry, opentelemetry-sdk, statsd, datadog
• Existing tracing: OTel configs (otel, tracing, OTEL_), jaeger, honeycomb, zipkin
• Existing health endpoints: /health, /healthz, /readiness, /liveness
• Deployment platform: fly.toml, Dockerfile, Kubernetes manifests, render.yaml, vercel.json
• Entrypoint file — where the app starts, so you know where to initialize OTel

Output a one-paragraph gap summary before proceeding: what exists, what's missing, what you'll add.

Step 1: Minimum Viable Instrumentation First

Before any custom spans or dashboards, establish the floor:

What goes in on day 1:

1. OTel SDK initialized at app startup, before any other imports
2. Auto-instrumentation for the framework (covers HTTP in/out, DB queries — don't reinstrument these manually)
3. Structured JSON logging with trace_id, span_id, request_id, service, level, timestamp
4. /healthz endpoint with dependency checks
5. OTLP export configured (or stdout in dev)

This is done before any custom instrumentation. It gets you RED metrics and traces with zero manual spans.

OTel initialization order matters. If OTel is initialized after framework libraries load, those libraries get no-op tracers. Always initialize first.

Language-specific bootstrap patterns

Node.js (Express/Fastify/Hapi):

// tracing.js — must be required FIRST via node -r ./tracing.js server.js
const { NodeSDK } = require("@opentelemetry/sdk-node");
const {
  getNodeAutoInstrumentations,
} = require("@opentelemetry/auto-instrumentations-node");
const {
  OTLPTraceExporter,
} = require("@opentelemetry/exporter-trace-otlp-http");
const {
  OTLPMetricExporter,
} = require("@opentelemetry/exporter-metrics-otlp-http");
const { PeriodicExportingMetricReader } = require("@opentelemetry/sdk-metrics");

const sdk = new NodeSDK({
  serviceName: process.env.OTEL_SERVICE_NAME || "my-service",
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();

Python (FastAPI/Flask/Django):

# otel_setup.py — import before anything else in main.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.auto_instrumentation import sitecustomize  # or use opentelemetry-instrument CLI

import os

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint=os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT")))
)
trace.set_tracer_provider(provider)

# Preferred: run via `opentelemetry-instrument python main.py`
# This auto-patches frameworks without code changes

Go:

// telemetry/setup.go
func InitOTel(ctx context.Context, serviceName string) (func(), error) {
    exporter, err := otlptracehttp.New(ctx)
    if err != nil { return nil, err }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceNameKey.String(serviceName),
        )),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{}, propagation.Baggage{},
    ))
    return func() { tp.Shutdown(ctx) }, nil
}
// Call in main() before http.ListenAndServe

Step 2: Structured Logging with Trace Correlation

Auto-instrumentation gives you traces. Now make logs queryable and correlatable.

Required fields on every log line: timestamp, level, message, service, trace_id, span_id, request_id

Node.js (pino):

const pino = require("pino");
const { trace, context } = require("@opentelemetry/api");

const logger = pino({ level: process.env.LOG_LEVEL || "info" });

function getLogger(req) {
  const span = trace.getActiveSpan();
  const ctx = span?.spanContext();
  return logger.child({
    service: process.env.OTEL_SERVICE_NAME,
    trace_id: ctx?.traceId,
    span_id: ctx?.spanId,
    request_id: req?.headers["x-request-id"],
  });
}

Python (structlog):

import structlog
from opentelemetry import trace

def add_otel_context(logger, method, event_dict):
    span = trace.get_current_span()
    if span.is_recording():
        ctx = span.get_span_context()
        event_dict["trace_id"] = format(ctx.trace_id, "032x")
        event_dict["span_id"] = format(ctx.span_id, "016x")
    return event_dict

structlog.configure(
    processors=[
        add_otel_context,
        structlog.processors.JSONRenderer(),
    ]
)

Do NOT log: PII, passwords, tokens, API keys, full request bodies, full response bodies.

Step 3: Custom Spans for Business-Critical Paths Only

Auto-instrumentation covers HTTP and DB. Add manual spans only where business context is missing — i.e., where you need to answer "which step of checkout failed?" not "which HTTP call failed?"

Add custom spans for:

• Multi-step business flows (checkout, onboarding, payment processing)
• External API calls that aren't HTTP (queue consumption, webhook processing)
• Cache logic that determines critical behavior
• Background jobs with meaningful SLAs

Do NOT add custom spans for:

• Individual DB queries (auto-instrumentation covers these)
• Simple helper functions
• Anything that adds < 1ms of latency and has no failure modes

Pattern (Node.js):

const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer("my-service");

async function processCheckout(cart) {
  return tracer.startActiveSpan("checkout.process", async (span) => {
    span.setAttributes({
      "checkout.item_count": cart.items.length,
      "checkout.total_cents": cart.totalCents,
      "user.id": cart.userId, // OK as span attribute, NOT as metric label
    });
    try {
      const result = await chargeCard(cart);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Use semantic conventions for attribute names (http.method, db.system, user.id) — don't invent names.

Step 4: Health Check Endpoint

Every service gets a /healthz endpoint. Keep it fast (< 200ms). Fail loudly on broken dependencies.

// Node.js example
app.get("/healthz", async (req, res) => {
  const checks = {};
  let healthy = true;

  // Check DB
  try {
    await db.query("SELECT 1");
    checks.database = "ok";
  } catch (e) {
    checks.database = "error";
    healthy = false;
  }

  // Check cache (non-critical — warn but don't fail)
  try {
    await redis.ping();
    checks.cache = "ok";
  } catch (e) {
    checks.cache = "degraded";
    // don't set healthy = false for non-critical deps
  }

  res.status(healthy ? 200 : 503).json({
    status: healthy ? "ok" : "error",
    checks,
    service: process.env.OTEL_SERVICE_NAME,
  });
});

If on Kubernetes or Cloud Run: wire /healthz to liveness and readiness probes. Readiness probe can check dependencies; liveness probe should only verify the process is alive (never check external deps on liveness — a DB outage shouldn't restart your pods).

Step 5: Export Configuration

Configure environment variables for the target platform. Prefer env vars over code — lets you change targets without deploys.

# .env.production — adjust OTLP endpoint per platform

# Grafana Cloud
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp-gateway-prod-us-central-0.grafana.net/otlp
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64-encoded-instance-id:api-key>

# Datadog
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.datadoghq.com
OTEL_EXPORTER_OTLP_HEADERS=DD-API-KEY=<api-key>

# Honeycomb
OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io
OTEL_EXPORTER_OTLP_HEADERS=x-honeycomb-team=<api-key>

# Self-hosted OTel Collector
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318

# All platforms
OTEL_SERVICE_NAME=my-service
OTEL_SERVICE_VERSION=1.2.3
OTEL_DEPLOYMENT_ENVIRONMENT=production

# Dev: dump to stdout
OTEL_TRACES_EXPORTER=console
OTEL_METRICS_EXPORTER=console

Sampling: 100% in dev and staging. Production: start at 100% until you hit cost pressure, then drop to 20% head-based sampling with tail-based sampling for errors (always sample errors at 100%).

Step 6: Output Summary

Follow the output format defined in docs/output-kit.md — 40-line CLI max, box-drawing skeleton, unified severity indicators.

## Instrumentation Summary

**Service:** [name]
**Stack:** [language / framework]
**Export target:** [platform]

### Added
- OTel SDK init: [where — entrypoint file]
- Auto-instrumentation: [what's covered — HTTP, DB, etc.]
- Structured logging: [library] — JSON with trace_id correlation
- Custom spans: [list of business flows instrumented, or "none needed"]
- Health check: /healthz — checks [list of dependencies]

### Skipped (intentional)
- [what was skipped and why — e.g., "no custom DB spans — auto-instrumentation covers queries"]

### Next step
- Define SLOs for this service, then run /vigil-alert to build alert rules