Help us improve
Share bugs, ideas, or general feedback.
From agentops-toolkit
Generate or update agentops.yaml (flat 1.0 schema) by inspecting the workspace. Trigger on "configure agentops", "agentops.yaml", "set up evaluation", "what should I evaluate". Infer the agent target and dataset from the codebase; ask only when nothing can be found.
npx claudepluginhub azure/agentops --plugin agentops-acceleratorHow this skill is triggered — by the user, by Claude, or both
Slash command
/agentops-toolkit:agentops-configThe summary Claude sees in its skill listing — used to decide when to auto-load this skill
Generate `agentops.yaml` at the project root. The flat schema has only a
Guides technical evaluation of code review feedback: read fully, restate for understanding, verify against codebase, respond with reasoning or pushback before implementing.
Share bugs, ideas, or general feedback.
Generate agentops.yaml at the project root. The flat schema has only a
handful of fields - most projects need just version, agent, and
dataset.
pip install "agentops-toolkit[foundry] @ git+https://github.com/Azure/agentops.git@develop" if agentops is missing.agentops eval analyze first. If it reports missing or ambiguous
target/dataset/scenario signals, use this skill to adapt the config.agentops.yaml does not exist, run agentops init first. The init
wizard already collects the agent reference and dataset path, so
agentops-config is most useful when the user wants to tweak an
existing config (add thresholds, switch to a different agent target,
add HTTP auth headers, etc.) rather than create one from scratch.Search the codebase for the strongest signal and pick one:
| Signal | agent: value |
|---|---|
AIProjectClient(...) + agent ID literal name:N | "<name>:<N>" |
Foundry hosted agent URL https://...services.ai.azure.com/...agents/... | the full URL |
| Any other HTTP endpoint your agent serves (FastAPI, Express, ACA, AKS) | the full URL |
Direct model use (openai.chat.completions.create(model=...)) with no orchestration | "model:<deployment-name>" |
Look in: README.md, main.py/server.py/app.ts, .env/.env.local,
.azure/<env>/.env, infra/, IaC outputs. If nothing is found, ask the
user once.
If a JSONL with rows that include input already exists in the repo, use
its path. Otherwise leave the default .agentops/data/smoke.jsonl and
hand off to the agentops-dataset skill before the first run.
Minimal example:
version: 1
agent: "my-rag:3"
dataset: .agentops/data/smoke.jsonl
HTTP/JSON example:
version: 1
agent: "https://my-aca-app.eastus2.azurecontainerapps.io/chat"
dataset: .agentops/data/smoke.jsonl
request_field: message # default is "message"
response_field: text # dot-path; default is "text"
auth_header_env: MY_API_TOKEN
Optional extras (only add when the user asks for them):
thresholds:
coherence: ">=3"
groundedness: ">=3"
avg_latency_seconds: "<=30"
# Publish results to the Foundry Evaluations panel.
# - execution: local + publish: true → Classic Foundry (uploads metrics)
# - execution: cloud → New Foundry (server-side run;
# publish is implicit, cloud always publishes)
execution: local
publish: true
# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"
# Cloud dataset submission policy. The local JSONL remains the source of truth;
# cloud runs sync it to Foundry Data/Datasets by default.
dataset_sync:
mode: auto # auto | foundry | inline
# name: agentops-smoke
# version: content-hash
evaluators: # rare - AgentOps auto-selects from agent + dataset
- name: similarity
threshold: ">=4"
Run agentops eval run once. If the config is malformed AgentOps prints a
clear error pointing at the offending key. Adjust and re-run.
bundle, target, execution, output,
backend). The 1.0 schema rejects them.dataset_sync.mode: auto
so AgentOps keeps Foundry Data/Datasets in sync; use inline only for quick
experiments or environments without dataset upload permission.