Create Scenario Eval

Create Scenario Eval

Guide the user through creating eval test cases for a Copilot Studio plugin scenario. Evals test end-to-end scenarios with natural prompts — the request routes through sub-agents (e.g., Author agent) which invoke skills internally.

How the eval system works

The eval harness (evals/evaluate.py) works by:

1. Copying a fixture agent into a temp workspace
2. Running claude -p "<prompt>" with a PreToolUse hook that traces skill invocations inside sub-agents
3. Checking routing (which agents and skills were invoked), output files, and response text against deterministic checks
4. Producing a JSON results file and HTML report

What can be tested right now

Authoring scenarios that produce YAML files (topics, agents, knowledge sources, etc.) are the best candidates. The harness supports these check types:

Check	What it validates	Use for
agent_invoked	Expected sub-agent was dispatched (e.g., Author agent)	Routing verification
agent_not_invoked	Unwanted sub-agents were NOT dispatched	Routing verification
skill_invoked	Expected skill was invoked (traced inside sub-agents via hook)	Skill routing
skill_not_invoked	Unwanted skills were NOT invoked	Skill routing
files_created	Expected files were created/modified (glob pattern)	All authoring scenarios
schema_validate	Full Copilot Studio schema validation (kind, required fields, IDs, Power Fx, scopes)	All YAML-producing scenarios
yaml_structure	Specific YAML path has expected value, min array length, or contains string	Structural assertions
content_contains	Keywords from prompt appear in output files	Domain relevance
no_placeholders	No _REPLACE, TODO, or FIXME markers left	Template completion
stdout_contains	CLI response text contains expected strings	Reference/info scenarios
stdout_not_contains	CLI response does NOT contain error strings	Error absence
exit_code	CLI exited with expected code	All scenarios
yaml_unchanged	Specific file or YAML path was NOT modified	Preservation testing

Note: no_placeholders runs automatically when any .mcs.yml file is changed, unless explicitly set to false.

Not yet testable: Integration scenarios that call external APIs (chat-directline, manage-agent) — these need script mocking which isn't implemented yet.

Available fixtures

Fixtures are pre-built agent directories in evals/fixtures/:

• basic-agent — Minimal agent with GenerativeActionsEnabled: false, one Greeting topic. Use for most authoring evals.
• agent-with-mcp-action — Same as basic-agent plus two MCP action files. Use for action-editing evals.
• empty-workspace — No agent files. Use for negative-path testing.

If the scenario needs a richer agent (e.g., existing topics to modify, knowledge sources, actions), note that the fixture would need to be created first.

Instructions

1. Identify the target scenario. If $ARGUMENTS is provided, use it as the scenario name. Otherwise ask the user what scenario they want to test (e.g., "topic creation", "agent settings", "knowledge sources").

2. Read relevant skill SKILL.md files to understand what the scenario covers:

   Glob: skills/*/SKILL.md
   

   Understand: What skills are involved? What YAML kinds? What files get created/modified?

3. Check if evals already exist:

   Glob: evals/scenarios/<scenario-name>.json
   

   If yes, read them and offer to add more test cases. Note the highest existing eval ID.

4. Guide the user through creating test cases. For each eval, gather:

   • name: Short descriptive title (e.g., "IT support topic with OnRecognizedIntent trigger")
   • prompt: A natural language prompt — what a real user would say. Do NOT prefix with "Use the X skill to...".
   • fixture: Which fixture agent to use (default: basic-agent)
   • checks: What to validate about the routing and output

5. Help the user define checks. Based on the scenario type:

   For topic-creation scenarios:

   {
     "agent_invoked": "copilot-studio:Copilot Studio Author",
     "skill_invoked": "copilot-studio:new-topic",
     "files_created": [{"pattern": "topics/*.topic.mcs.yml", "min_count": 1}],
     "schema_validate": true,
     "yaml_structure": [
       {"path": "kind", "equals": "AdaptiveDialog"},
       {"path": "beginDialog.kind", "equals": "<trigger-type>"}
     ],
     "content_contains": ["<domain keywords>"],
     "no_placeholders": true
   }
   

   For agent-settings scenarios:

   {
     "agent_invoked": "copilot-studio:Copilot Studio Author",
     "skill_invoked": "copilot-studio:edit-agent",
     "files_created": [{"pattern": "agent.mcs.yml", "min_count": 1}],
     "schema_validate": true,
     "yaml_structure": [
       {"path": "kind", "equals": "GptComponentMetadata"}
     ],
     "content_contains": ["<expected content>"],
     "no_placeholders": true
   }
   

   For knowledge-source scenarios:

   {
     "agent_invoked": "copilot-studio:Copilot Studio Author",
     "skill_invoked": "copilot-studio:add-knowledge",
     "files_created": [{"pattern": "knowledge/*.knowledge.mcs.yml", "min_count": 1}],
     "schema_validate": true,
     "no_placeholders": true
   }
   

   For reference/query scenarios:

   {
     "stdout_contains": ["<expected content in response>"],
     "exit_code": 0
   }
   

6. Recommend at least 3 test cases that cover different possibilities within the scenario. For example, for topic-creation:

   • Different trigger types (OnRecognizedIntent, OnConversationStart, OnUnknownIntent)
   • Different complexity levels (simple message, multi-step with questions, branching)
   • Edge cases (empty workspace refusal)

7. Write the scenario JSON file:

   Write: evals/scenarios/<scenario-name>.json
   

   Format:

   {
     "scenario_name": "<scenario-name>",
     "evals": [
       {
         "id": 1,
         "name": "<short descriptive title>",
         "prompt": "<natural language request — what a user would say>",
         "fixture": "basic-agent",
         "mock_scripts": [],
         "checks": { ... }
       }
     ]
   }
   

8. Tell the user how to run the evals:

   python3 evals/evaluate.py --scenario <scenario-name> --verbose
   

   Or for all scenarios: node evals/run.js

   To generate the HTML report: python3 evals/report.py evals/results/<timestamp>/

Important guidelines

• Prompts must be natural language — write what a real user would say, not "Use the X skill to..."
• Include agent_invoked and skill_invoked checks to verify correct routing
• Keep prompts specific enough that checks can be deterministic (mention exact names, values, counts)
• Use schema_validate: true for ALL scenarios that produce YAML — it's the most powerful check
• content_contains keywords should come directly from the prompt to verify domain relevance
• Don't create evals for deprecated skills (chat-with-agent, directline-chat)
• Eval IDs must be unique integers within a scenario's JSON