llm-observability

The tooling for watching LLM apps in production is a fast-moving mess of overlapping projects, and every list I found had stars from 2023 on it. So I kept my own, verified the entries, and wired up a job to keep the numbers honest. Use it to find the right tool without re-researching the whole space.

Legend: 🟢 open-source · 🔵 open-core / hybrid · 🟠 commercial (public repo is an SDK/client only - low star counts don't reflect the product). Star counts are pulled live from the GitHub API and auto-refreshed weekly by CI (tools/refresh_stars.py), so they stay current instead of rotting.

Contents

• What is LLM Observability?
• 🧰 Skills (Batteries Included)
• Platform Comparison (At a Glance)
• Tracing & Observability Platforms
• Evaluation Frameworks
• Prompt Management & Experimentation
• LLM Gateways & Proxies
• Instrumentation & Standards
• Guardrails & Safety Monitoring
• Self-Hosted / Open-Source First
• Research & Benchmarks
• Learning Resources
• Guides & Tools (Original)
• How to Choose
• Contributing

What is LLM Observability?

Regular observability assumes your system is deterministic. LLM apps aren't. They make things up, drift as inputs change, quietly burn tokens, and fail without ever throwing an error. So you end up watching different things:

• tracing: every step of a run, with tokens, latency, and cost per span
• evals: whether the output is actually any good, both offline and on live traffic
• monitoring: alerts when quality, cost, latency, or error rate move
• prompt and dataset management: version your prompts, and pull real production cases back into your tests

People call this LLMOps observability, AI observability, or just LLM monitoring. Same idea.

🧰 Skills (Batteries Included)

Most lists stop at links. This one also ships 26 agent skills, the SKILL.md kind, so your coding agent knows how to actually do this stuff instead of just knowing where the tools live. They fire on plain requests like "add tracing" or "reduce my LLM bill".

A handful (marked ⚙️) are backed by real Python with unit tests that run in CI. The rest ship trigger cases and get checked by skillvitals, a small tool I built to measure whether a skill actually fires when it should. A free heuristic runs on every push as a regression gate.

Graded on a cheap model (llama-3.1-8b via NVIDIA NIM), the 26 skills average a trigger F1 of 0.99 (98.7% recall, 0% false-fire), and that holds even against adjacent negatives, prompts about a different LLM task the skill should refuse to fire on. It's not a clean sweep though. choose-observability-stack came in at 0.80 because the model kept confusing a "compare Langfuse vs Phoenix" prompt with the model-comparison skill. That's exactly the kind of overlap this is meant to catch. Run it yourself:

skillvitals trigger skills/reduce-llm-cost/SKILL.md \
  --cases validation/cases/reduce-llm-cost.yaml \
  --model meta/llama-3.1-8b-instruct --base-url https://integrate.api.nvidia.com/v1