BigQuery Agent Analytics SDK
An open-source Python SDK for analyzing, evaluating, and curating agent traces
stored in BigQuery. Built on top of the
BigQuery Agent Analytics, it provides
a consumption-layer toolkit for agent observability, analysis, evaluation, and advanced capabilities like context graph at scale.
Overview
The BigQuery Agent Analytics SDK connects your AI agent telemetry in BigQuery to
a rich set of evaluation, observability, and analytics capabilities. It is
designed for ML engineers, data scientists, and platform teams who run agents in
production and need to understand agent behavior, measure quality, and detect
regressions — all through BigQuery SQL or Python.
Key Features
Observability
- Trace reconstruction and DAG visualization
- Per-event-type BigQuery views
- Observability dashboards (SQL and BigFrames)
Evaluation
- Code-based metrics (latency, turn count, error rate, token efficiency, cost)
- LLM-as-Judge scoring (correctness, hallucination, sentiment)
- Trajectory matching (exact, in-order, any-order)
- Multi-trial evaluation with pass@k / pass^k
- Grader composition (weighted, binary, majority strategies)
- Eval suite lifecycle management with graduation and saturation detection
- Static quality validation (ambiguous tasks, class imbalance, suspicious thresholds)
AI/ML Integration
- BigQuery AI.GENERATE, AI.EMBED, AI.CLASSIFY
- Anomaly detection and latency forecasting
- Categorical (Hatteras-style) evaluation via BigFrames
Advanced Analytics
- Context Graph — property graph linking traces to business entities with GQL traversal
- YAML-driven ontology extraction and materialization
- Long-horizon cross-session memory
- Multi-stage agent insights pipeline
- Drift detection for golden vs production question distributions
CLI (bq-agent-sdk)
- 12+ commands for diagnostics, evaluation, and CI/CD integration
Deployment Surfaces
- Remote Function (BigQuery SQL via Cloud Run)
- Python UDF scoring kernels
- Streaming evaluation (Cloud Scheduler + Cloud Run)
- Continuous query templates
Usage Telemetry
- Every job the SDK submits is labeled (
sdk, sdk_version,
sdk_surface, sdk_feature, and sdk_ai_function where relevant)
so operators can attribute spend, latency, and adoption directly
from INFORMATION_SCHEMA.JOBS_BY_PROJECT. No extra telemetry
pipeline is required. See docs/sdk_usage_tracking.md
for the label schema and ready-to-run tracking queries.
Prerequisites
Installation
pip install bigquery-agent-analytics
With optional LLM judge support:
pip install bigquery-agent-analytics[llm]
With BigFrames support:
pip install bigquery-agent-analytics[bigframes]
Quick Start
from bigquery_agent_analytics import Client
client = Client(project_id="my-project", dataset_id="analytics")
trace = client.get_trace("trace-abc-123")
trace.render()
See SDK.md for the full API walkthrough with code examples for every
feature.
Documentation
Architecture
