agentops-accelerator

AgentOps Accelerator

Evaluate. Ship. Observe. Operate.
Continuous evaluation, safety testing, observability, and release readiness for Microsoft Foundry agents.

Documentation | PyPI | VS Code Extension | Latest release

AgentOps Accelerator helps Microsoft Foundry agent teams evaluate quality, prepare releases, monitor behavior, and operate reliably after launch. It gives you a practical starting point for agent operations, with Foundry integration as the default path and deeper setup guidance in the full docs.

Get started

python -m pip install agentops-accelerator
agentops init

agentops init starts a guided setup that creates your agentops.yaml and .agentops/ workspace.

Next, follow the tutorial that matches your agent type:

• Prompt Agent tutorial
• Hosted or HTTP Agent tutorial

What it helps you do

Use AgentOps Accelerator when you need to:

• Evaluate an agent before release
• Compare changes across versions
• Capture release evidence
• Monitor agent quality and regressions
• Give teams a repeatable way to operate agents responsibly in production

The accelerator keeps the local workflow simple, then points you to the full docs when you are ready to configure pipelines, dashboards, and release practices.

Learn more

For setup guides, tutorials, architecture, CI/CD guidance, Doctor checks, and evaluator reference, start with the documentation site:

https://aka.ms/agentops-accelerator

Run a first evaluation

az login
$env:AZURE_AI_FOUNDRY_PROJECT_ENDPOINT = "https://<resource>.services.ai.azure.com/api/projects/<project>"
$env:AZURE_OPENAI_ENDPOINT = "https://<openai-resource>.openai.azure.com"
$env:AZURE_OPENAI_DEPLOYMENT = "gpt-4o-mini"
agentops eval analyze
agentops eval run
agentops doctor --evidence-pack

For Foundry targets, use either project_endpoint: in agentops.yaml or AZURE_AI_FOUNDRY_PROJECT_ENDPOINT. Config wins when both are set.

Outputs land in .agentops/results/latest/:

• results.json - machine-readable (versioned, stable schema)
• report.md - human-readable, PR-friendly

Release evidence lands in .agentops/release/latest/:

• evidence.json - machine-readable production-readiness projection
• evidence.md - PR/release summary

Capture the first successful run as a baseline:

New-Item -ItemType Directory -Force .agentops\baseline | Out-Null
Copy-Item .agentops\results\latest\results.json .agentops\baseline\results.json

To see a visible comparison, publish a new agent version with a prompt that paraphrases instead of copying exact-answer requests, update agentops.yaml to that new name:version, and compare against the baseline:

agentops eval run --baseline .agentops/baseline/results.json

The report grows a Comparison vs Baseline section with per-metric deltas.

---

Commands

Install optional extras as needed: [agent] for Doctor/Cockpit and [mcp] for MCP.