tensorzero

TensorZero

TensorZero is an open-source LLMOps platform that unifies:

• Gateway: access every LLM provider through a unified API, built for performance (<1ms p99 latency)
• Observability: store inferences and feedback in your database, available programmatically or in the UI
• Evaluation: benchmark individual inferences or end-to-end workflows using heuristics, LLM judges, etc.
• Optimization: collect metrics and human feedback to optimize prompts, models, and inference strategies
• Experimentation: ship with confidence with built-in A/B testing, routing, fallbacks, retries, etc.

You can take what you need, adopt incrementally, and complement with other tools. It plays nicely with the OpenAI SDK, OpenTelemetry, and every major LLM provider.

TensorZero is used by companies ranging from frontier AI startups to the Fortune 10 and fuels ~1% of global LLM API spend today.

Website · Docs · Twitter · Slack · Discord

Quick Start (5min) · Deployment Guide · API Reference · Configuration Reference

Demo

Features

[!NOTE]

🆕 TensorZero Autopilot

TensorZero Autopilot is an automated AI engineer powered by TensorZero that analyzes LLM observability data, sets up evals, optimizes prompts and models, and runs A/B tests.

It dramatically improves the performance of LLM agents across diverse tasks:

Learn more →  Schedule a demo →

🌐 LLM Gateway

Integrate with TensorZero once and access every major LLM provider.

• Call any LLM (API or self-hosted) through a single unified API
• Infer with tool use, structured outputs (JSON), batch, embeddings, multimodal (images, files), caching, etc.
• Create prompt templates and schemas to enforce a structured interface between your application and the LLMs
• Satisfy extreme throughput and latency needs, thanks to 🦀 Rust: <1ms p99 latency overhead at 10k+ QPS
• Ensure high availability with routing, retries, fallbacks, load balancing, granular timeouts, etc.
• Track usage and cost and enforce custom rate limits with granular scopes (e.g. tags)
• Set up auth for TensorZero to allow clients to access models without sharing provider API keys

Supported Model Providers