Skill

phoenix-evals

Builds and runs evaluators for AI/LLM apps using Phoenix in Python or TypeScript. Covers code/LLM judges, batch eval, experiments, datasets, validation, and production.

Python

Typescript

Install

Run in your terminal

npx claudepluginhub ctr26/dotfiles --plugin awesome-copilot

Tool Access

This skill uses the workspace's default tool permissions.

Supporting Assets

View in Repository

references/axial-coding.md

references/common-mistakes-python.md

references/error-analysis-multi-turn.md

references/error-analysis.md

references/evaluate-dataframe-python.md

references/evaluators-code-python.md

references/evaluators-code-typescript.md

references/evaluators-custom-templates.md

references/evaluators-llm-python.md

references/evaluators-llm-typescript.md

references/evaluators-overview.md

references/evaluators-pre-built.md

references/evaluators-rag.md

references/experiments-datasets-python.md

references/experiments-datasets-typescript.md

references/experiments-overview.md

references/experiments-running-python.md

references/experiments-running-typescript.md

references/experiments-synthetic-python.md

references/experiments-synthetic-typescript.md

Skill Content

Similar Skills

context7-mcp

Fetches up-to-date documentation from Context7 for libraries and frameworks like React, Next.js, Prisma. Use for setup questions, API references, and code examples.

context7-plugin

51.8k

context7-mcp

Fetches up-to-date documentation from Context7 for libraries and frameworks like React, Next.js, Prisma. Use for setup questions, API references, and code examples.

context7

50.8k

context7-cli

3 files

Uses ctx7 CLI to fetch current library docs, manage AI coding skills (install/search/generate), and configure Context7 MCP for AI editors.

context7

50.8k

Stats

Stars28207

Forks3270

Last CommitApr 1, 2026

Actions

View Source View Plugin View on GitHub View README

Phoenix Evals

Build evaluators for AI/LLM applications. Code first, LLM for nuance, validate against humans.

Quick Reference

Task	Files
Setup	setup-python, setup-typescript
Decide what to evaluate	evaluators-overview
Choose a judge model	fundamentals-model-selection
Use pre-built evaluators	evaluators-pre-built
Build code evaluator	evaluators-code-python, evaluators-code-typescript
Build LLM evaluator	evaluators-llm-python, evaluators-llm-typescript, evaluators-custom-templates
Batch evaluate DataFrame	evaluate-dataframe-python
Run experiment	experiments-running-python, experiments-running-typescript
Create dataset	experiments-datasets-python, experiments-datasets-typescript
Generate synthetic data	experiments-synthetic-python, experiments-synthetic-typescript
Validate evaluator accuracy	validation, validation-evaluators-python, validation-evaluators-typescript
Sample traces for review	observe-sampling-python, observe-sampling-typescript
Analyze errors	error-analysis, error-analysis-multi-turn, axial-coding
RAG evals	evaluators-rag
Avoid common mistakes	common-mistakes-python, fundamentals-anti-patterns
Production	production-overview, production-guardrails, production-continuous

Workflows

Starting Fresh: observe-tracing-setup → error-analysis → axial-coding → evaluators-overview

Building Evaluator: fundamentals → common-mistakes-python → evaluators-{code|llm}-{python|typescript} → validation-evaluators-{python|typescript}

RAG Systems: evaluators-rag → evaluators-code-* (retrieval) → evaluators-llm-* (faithfulness)

Production: production-overview → production-guardrails → production-continuous

Reference Categories

Prefix	Description
`fundamentals-*`	Types, scores, anti-patterns
`observe-*`	Tracing, sampling
`error-analysis-*`	Finding failures
`axial-coding-*`	Categorizing failures
`evaluators-*`	Code, LLM, RAG evaluators
`experiments-*`	Datasets, running experiments
`validation-*`	Validating evaluator accuracy against human labels
`production-*`	CI/CD, monitoring

Key Principles

Principle	Action
Error analysis first	Can't automate what you haven't observed
Custom > generic	Build from your failures
Code first	Deterministic before LLM
Validate judges	>80% TPR/TNR
Binary > Likert	Pass/fail, not 1-5