OTel Tracing

Create spans for logical operations, propagate context across every boundary, use semantic conventions for attribute names. Every tracing decision — span granularity, attribute selection, sampling strategy — trades off cost against visibility.

References

Topic	Reference	Contents
Spans	references/spans.md	Span anatomy, root spans, lifetime code patterns
Span data	references/span-data.md	Events format, links format, SDK limits table
Context propagation	references/context-propagation.md	W3C header format, propagator selection, baggage details, security
Instrumentation	references/instrumentation.md	Server/client/async code patterns, library rules, testing guidance
Sampling	references/sampling.md	Head/tail/combined strategies, decision guide, sampler types
Semantic conventions	references/semantic-conventions.md	HTTP/DB/messaging attribute lists, status mapping, general conventions
SDK components	references/sdk-components.md	Resource config, env vars, Collector deployment, exporter types

Spans

A span represents a single unit of work in a trace — an HTTP request handler, a database query, a message publish. Not every function call.

Span Naming

Name spans for the class of operation, not the instance. Low-cardinality names enable aggregation; high-cardinality names destroy it.

• Use the most general string that identifies a statistically interesting class
• Never embed high-cardinality values (IDs, emails, paths with IDs)
• HTTP span names: {METHOD} {route} — server: GET /users/:id; client: GET
• Database span names: {operation} {table} — SELECT users, INSERT orders

Name	Verdict
get	Too general
get_account/42	Too specific — ID in name
get_account	Good
GET /users/{userId}	Good — route template
POST /api/v2/orders/abc123	Bad — order ID in name

SpanKind

SpanKind tells backends how to assemble the trace tree. Set it correctly.

Kind	Direction	Use For
SERVER	Incoming	HTTP handler, gRPC server method
CLIENT	Outgoing	HTTP client call, DB query, gRPC call
PRODUCER	Outgoing	Enqueue message, schedule job
CONSUMER	Incoming	Dequeue message, process job
INTERNAL	Neither	In-process business logic, computation

• A single span MUST NOT serve more than one purpose
• Create a new span before injecting context for outgoing calls
• CLIENT -> SERVER for synchronous calls; PRODUCER -> CONSUMER for async
• Using INTERNAL for HTTP handlers or DB calls is wrong — use SERVER or CLIENT

Span Status

Status	Meaning	When to Set
Unset	No error	Default — do not change on success
Error	Operation failed	When an error occurs; include description
Ok	Explicitly successful	Only to override a previous Error; rarely needed

• Leave status Unset on success — it already means "no error"
• Ok is a "final call" — once set, subsequent Error attempts are ignored
• Instrumentation libraries SHOULD NOT set Ok — leave to application code

Span Lifetime

• Every created span MUST be ended — leaked spans cause memory issues and incomplete traces
• After end(), the span becomes non-recording; further mutations are ignored
• Use language-idiomatic patterns: defer (Go), try/finally (Java/JS), context manager (Python)

Span Data: Attributes, Events, Links

Attributes

Key-value pairs annotating a span. Value types: string, boolean, integer, float, or arrays of these.

• Use semantic conventions for attribute names — http.request.method, db.system, not custom names. Consistent naming enables cross-service analysis.
• Add sampling-relevant attributes at span creation — samplers can only see attributes present at creation time.
• Keep cardinality low — attribute values should be from a bounded set.
• Check IsRecording before expensive attribute computation — sampled-out spans discard all data.
• Never store PII in attributes — trace data flows to shared backends. Use opaque identifiers.
• Respect attribute limits — SDK enforces AttributeCountLimit (default 128). Excess attributes are silently dropped.

Events

Timestamped annotations — structured log entries attached to a span.

• Use events when the timestamp matters; use attributes for metadata with no meaningful timestamp
• Add events for key domain occurrences ("page became interactive", "retry attempted")
• Use events for verbose data instead of additional spans

Recording Exceptions

• Always pair RecordException with setStatus(ERROR) — RecordException alone does not change span status
• Never swallow application exceptions in instrumentation — catch instrumentation errors, log them, but always rethrow application exceptions

Links

Associate a span with spans from other traces without parent-child hierarchy.

• Batch processing: link consumer span to each producer span
• Async follow-up: job span links back to triggering span
• Add links at span creation when possible — samplers can consider them

See references/span-data.md for events format, links format, and SDK limits table.

Context Propagation

Context propagation correlates spans across service boundaries. Without it, each service produces isolated spans — no distributed trace.

Inject / Extract Pattern

1. Sender creates a span, makes it current, injects context into outgoing carrier (HTTP headers, message metadata)
2. Receiver extracts context from incoming carrier and uses it as parent for new spans

• Call inject AFTER creating the outgoing CLIENT or PRODUCER span
• Call extract BEFORE creating the server/consumer span
• Use W3C TraceContext as the default propagation format

In-Process Propagation

• Make spans active/current — enables automatic log correlation, nested auto-instrumentation picking up correct parent, context propagation to child ops
• Pass context explicitly in async code — automatic propagation may break across thread boundaries, goroutines, or async callbacks
• Never propagate request context to background work — background tasks should start new traces and link back to the triggering span

Baggage

Propagates arbitrary key-value pairs across service boundaries alongside trace context.

• Use for: tenant ID, request priority, feature flags, sampling hints
• NEVER put PII, credentials, or API keys in baggage — visible to all downstream services

See references/context-propagation.md for W3C header format, propagator selection table, baggage details, and security considerations.

Instrumentation

Approaches

Approach	When	Trade-off
Automatic	Supported libraries (HTTP, DB, messaging)	Zero code, no business attrs
Manual	Business logic, unsupported libraries	Full control, more code
Hybrid (recommended)	Production	Auto for infra + manual for business logic

Getting a Tracer

• Name the tracer after your library, package, or module — not the application
• Include a version string matching your library version
• Libraries: accept TracerProvider via dependency injection or use the global one
• Applications: configure the SDK and set the global TracerProvider

What to Instrument

Good candidates: public API methods with I/O or significant computation, request/message handlers, outbound calls (HTTP, DB, RPC), background jobs.

Poor candidates: every function call (noise), thin wrapper libraries (already instrumented underneath), pure computation with no I/O and sub-ms duration.

Decision: Is it a network call or significant I/O? -> Create span (CLIENT/SERVER). Is it a meaningful business operation? -> Create span (INTERNAL). Would a span event on parent suffice? -> Add event. Otherwise -> don't instrument.

Library Instrumentation Rules

• Depend on OpenTelemetry API only — never SDK. The API is a no-op without SDK, so zero overhead for users who don't use OTel.
• Follow semantic conventions for your domain
• Set the schema_url to record which semantic convention version you use
• Prefer events over spans for verbose internal details
• Support optional TracerProvider injection for testability

See references/instrumentation.md for server-side, client-side, and async producer/consumer patterns, plus testing guidance.

Sampling

Sampling controls which traces are recorded and exported — the primary mechanism for managing tracing costs.

Head Sampling

Decision at trace creation time, before any spans complete.

Sampler	Behavior
AlwaysOn	Record and sample everything
AlwaysOff	Drop everything
TraceIdRatioBased(ratio)	Sample based on trace ID hash
ParentBased(root)	Delegate based on parent sampling decision

ParentBased is the most common production configuration — respects parent decisions, applies custom root sampling. Default SDK sampler: ParentBased(root=AlwaysOn).

Tail Sampling

Decision after all spans in a trace complete. Requires collector infrastructure.

• All services export at 100%
• OTel Collector with tail sampling processor buffers spans by trace ID
• All spans with the same trace ID MUST reach the same collector — use trace-ID-aware load balancing

Sampling Principles

• Start with no sampling — add only when cost or volume requires it
• Always sample errors — configure tail sampling to keep error traces
• Use ParentBased — children follow parent decisions for complete traces
• Provide attributes at span creation — samplers cannot see late-added attributes
• Filter health checks — high volume, low value; sample aggressively or filter entirely

See references/sampling.md for combined head+tail strategies, decision guide, and per-scenario recommendations.

Semantic Conventions

Use semantic conventions for span names and attributes. Consistent naming enables cross-service analysis without learning custom attribute names.

• HTTP server spans: kind SERVER, name {METHOD} {http.route}, never full URI path
• HTTP client spans: kind CLIENT, name {METHOD}
• Database spans: kind CLIENT, name {operation} {target}, always sanitize queries
• Messaging producer: kind PRODUCER, name {destination} publish
• Messaging consumer: kind CONSUMER, name {destination} process

See references/semantic-conventions.md for required/recommended attributes per domain, status mapping rules, and general conventions.

SDK Components

TracerProvider

• Initialize once, early in application startup
• Always call shutdown() on exit — flushes remaining spans
• Configure Resource with service.name — identifies your service in backends
• Use BatchSpanProcessor in production — SimpleSpanProcessor is for dev/testing only

See references/sdk-components.md for Resource attributes, BatchSpanProcessor tuning, exporter types, environment variable configuration, and Collector deployment patterns.

Application

When writing tracing code:

• Apply all conventions silently — don't narrate each rule being followed.
• Use semantic conventions for attribute names. Check the reference for your domain (HTTP, DB, messaging) before inventing names.
• If an existing codebase contradicts a convention, follow the codebase and flag the divergence once.

When reviewing tracing code:

• Cite the specific violation and show the fix inline.
• Focus on: span naming cardinality, missing context propagation, missing error recording, wrong SpanKind, missing span.end().

Integration

The coding skill governs workflow; this skill governs tracing implementation choices. Language-specific skills handle SDK API differences.