instrumenting-with-mlflow-tracing

MLflow Tracing Instrumentation Guide

Language-Specific Guides

Based on the user's project, load the appropriate guide:

• Python projects: Read references/python.md
• TypeScript/JavaScript projects: Read references/typescript.md

If unclear, check for package.json (TypeScript) or requirements.txt/pyproject.toml (Python) in the project.

---

What to Trace

Trace these operations (high debugging/observability value):

Operation Type	Examples	Why Trace
Root operations	Main entry points, top-level pipelines, workflow steps	End-to-end latency, input/output logging
LLM calls	Chat completions, embeddings	Token usage, latency, prompt/response inspection
Retrieval	Vector DB queries, document fetches, search	Relevance debugging, retrieval quality
Tool/function calls	API calls, database queries, web search	External dependency monitoring, error tracking
Agent decisions	Routing, planning, tool selection	Understand agent reasoning and choices
External services	HTTP APIs, file I/O, message queues	Dependency failures, timeout tracking

Skip tracing these (too granular, adds noise):

• Simple data transformations (dict/list manipulation)
• String formatting, parsing, validation
• Configuration loading, environment setup
• Logging or metric emission
• Pure utility functions (math, sorting, filtering)

Rule of thumb: Trace operations that are important for debugging and identifying issues in your application.

---

Verification

After instrumenting the code, always verify that tracing is working.

Planning to evaluate your agent? Tracing must be working before you run agent-evaluation. Complete verification below first.

1. Run the instrumented code — execute the application or agent so that at least one traced operation fires
2. Confirm traces are logged — use mlflow.search_traces() or MlflowClient().search_traces() to check that traces appear in the experiment:

import mlflow

traces = mlflow.search_traces(experiment_ids=["<experiment_id>"])
print(f"Found {len(traces)} trace(s)")
assert len(traces) > 0, "No traces were logged — check tracking URI and experiment settings"

3. Verify spans were captured — confirm the trace contains the expected spans, not just an empty shell:

trace = traces.iloc[0]
spans = mlflow.get_trace(trace.trace_id).data.spans
print(f"Trace has {len(spans)} span(s)")
for span in spans:
    print(f"  - {span.name} ({span.span_type})")

4. Report the result — tell the user how many traces and spans were found and confirm tracing is working

If no traces appear

Check these in order:

• Tracking URI not set — is mlflow.set_tracking_uri(...) called before the agent run? Without this, traces go to a local ./mlruns directory instead of the configured server.
• Autolog warnings — did mlflow.autolog() or framework-specific mlflow.<framework>.autolog() raise any warnings during setup? Check stderr for patching failures.
• Wrong experiment ID — verify the experiment ID passed to search_traces() matches the experiment active when the code ran (mlflow.get_experiment_by_name(...) to confirm).
• Network/auth issues — can the process reach the tracking server? Check for connection errors or 401/403 responses in logs.

For automated validation, use agent-evaluation/scripts/validate_tracing_runtime.py.

---

Feedback Collection

Log user feedback on traces for evaluation, debugging, and fine-tuning. Essential for identifying quality issues in production.

See references/feedback-collection.md for:

• Recording user ratings and comments with mlflow.log_feedback()
• Capturing trace IDs to return to clients
• LLM-as-judge automated evaluation

---

Reference Documentation

Production Deployment

See references/production.md for:

• Environment variable configuration
• Async logging for low-latency applications
• Sampling configuration (MLFLOW_TRACE_SAMPLING_RATIO)
• Lightweight SDK (mlflow-tracing)
• Docker/Kubernetes deployment

Advanced Patterns

See references/advanced-patterns.md for:

• Async function tracing
• Multi-threading with context propagation
• PII redaction with span processors

Distributed Tracing

See references/distributed-tracing.md for:

• Propagating trace context across services
• Client/server header APIs