copilot-studio-bot-design

Copilot Studio Bot Design Skill

Purpose: Senior-maker playbook for copilot-studio-engineer covering the architecture decisions that determine whether a Copilot Studio bot is a useful coworker or a hallucinating embarrassment in front of customers. The decisions are made on day one — topic vs generative-answers boundaries, knowledge sources, escalation criteria — and the test discipline that keeps them honest as the bot evolves.

When to Use

• Starting a new bot — before the first topic is built, work through the architecture decisions here.
• A deployed bot is misbehaving — hallucinating answers, dropping conversations, escalating too often or not often enough, ignoring intent. This skill covers the diagnostic playbook.
• Adding a new knowledge source or topic to an existing bot — verify it doesn't blow up the existing test set.
• Deciding AI implementation path — Copilot Studio vs AI Builder vs prompt-flow vs direct Azure OpenAI. The trade-off matters for cost, governance, and freshness.

Core Principles

1. A bot is a contract about what the org will answer, not a wrapper around an LLM. Treat it like a product surface — versioned, tested, owned.
2. Topics for high-stakes, regulated, or specific-format answers. Generative-answers for everything else. The line is "would I want a human to read this exact response before it shipped?" If yes, it's a topic.
3. Knowledge-source hygiene is the #1 hallucination prevention. Garbage in, confidently-wrong out. Stale, duplicated, or contradictory sources poison the bot worse than no source.
4. Every authoring change re-runs the regression set. A 20-prompt test set is the difference between "we improved the bot" and "we hope we improved the bot."
5. Escalation to human is a feature, not a failure. Bots that never escalate are usually hallucinating instead of saying "I don't know."
6. Premium connector cost is real — Copilot Studio messages are licensed; AI Builder credits are counted; direct Azure OpenAI is metered. Every architecture choice has a unit-economics consequence.

Playbook

1. Copilot Studio architecture in plain English

A Copilot Studio bot has four kinds of building blocks:

• Topics — authored conversations with explicit trigger phrases, slot-filling, branching, and exit conditions. Deterministic. You wrote it; the bot says exactly what you wrote.
• Generative answers — retrieval-augmented generation (RAG) over the bot's knowledge sources. The bot reads from sources at query time, an LLM composes the answer. Non-deterministic but grounded.
• Knowledge sources — websites, SharePoint, documents, Dataverse, custom data — pointed at the bot, indexed, queryable by generative answers.
• Plugins / actions — calls into Power Automate flows, custom connectors, AI Builder prompts, or external APIs. Executes work; not just speech.

The bot router decides per-utterance: matches an authored topic? Run it. Otherwise hand to generative answers, or escalate.

2. Topic vs generative-answers — the boundary

Use a topic when…	Use generative answers when…
Regulated content (legal, HR policy, financial disclosure)	Informational FAQ over public-ish content
High-blast-radius actions (file a claim, place an order, transfer money)	Synthesizing across many docs to answer a question
The answer has a specific format (table, JSON, exact wording)	The answer space is broad and slow-changing
Slot-filling required (you need 5 fields to do anything)	The user's question is usually one-shot
Compliance demands logged determinism	Variation in phrasing is acceptable

Test for the line: "if a user gets a wrong answer here, who explains it to compliance / legal / the customer?" If the answer is a real person, it should be a topic.

3. Knowledge-source hygiene

The most common cause of bot hallucination is bad sources, not bad models.

• Curate over collect. Pointing the bot at the whole intranet is a recipe for stale, contradictory answers. Pick canonical sources per topic area.
• Publish-to-bot vs reference. Some sources should be authoritative; others should be reference-only. Mark them accordingly so the bot can cite confidence-weighted.
• Freshness control. Each source has a refresh interval. If a SharePoint page changes daily but indexes weekly, users see stale answers. Set the cadence deliberately.
• De-duplicate and resolve contradictions. If three pages explain the same policy differently, the bot will pick one — often the wrong one. Fix the source content before adding more.
• Test each source in isolation. Add it, run a topic-specific test set, verify no regression.

4. Trigger-phrase design

Trigger phrases are intent classes, not synonyms. Each topic should have 8-15 trigger phrases representing the range of how users would express that intent.

• Bad: "reset password", "reset my password", "password reset" (3 variants of one phrase)
• Good: "reset password", "forgot password", "can't log in", "locked out", "need to change my password", "my password isn't working", "request a new password" (range of intent expressions)

The Copilot Studio NLU model generalises from the range. Narrow training data = brittle triggering.

5. Slot-filling patterns

For any topic that requires inputs:

• Required vs optional — explicitly mark. Optional slots have a default.
• Validation — every slot has a validation expression. "Email" requires regex; "Order Number" requires format check; "Date" requires date parse.
• Re-prompt — when validation fails, the re-prompt should explain why it failed and give an example. "I didn't understand — could you provide your order number in the format ABC-12345?"
• Maximum attempts — after 3 failed attempts, escalate. Looping forever frustrates users and hides bugs.
• Confirmation — before any irreversible action, show the user the collected values and ask "is this right?" Slot-filling without confirmation is how the wrong record gets updated.

6. Guardrails (system prompt, content moderation, off-topic redirection)

• System prompt at the bot level establishes voice, scope, and refusal patterns. "You are a customer service assistant for ACME. You only answer questions related to ACME products and orders. For anything else, politely redirect to live chat."
• Content moderation — Copilot Studio has built-in moderation; verify it's on. For sensitive domains, layer Purview labels and a custom moderation flow.
• Off-topic redirection — when generative answers return low-confidence or the user asks about an obvious off-topic (weather, news, jokes), respond with the redirection pattern, not an attempt.
• Hallucination guardrails — turn on "moderate" or "strict" grounding mode for generative answers; raise the confidence threshold; require citations.

7. Escalation to human — when, how, context

• When: explicit user request ("talk to a person"), N failed slot-fill attempts, low-confidence generative answer in a sensitive topic, any topic explicitly marked "always escalate" (legal, complaints, billing disputes).
• How: Teams channel handoff, Dynamics Customer Service queue, third-party live-chat integration, or email-the-conversation. Pick one and instrument it.
• Context handoff: the human picks up with the full conversation transcript, the detected intent, the values collected in slots, and any actions the bot took. A handoff with no context is worse than no bot — the user has to re-explain everything.

8. AI Builder vs prompt-flow vs direct Azure OpenAI

Decision matrix for "where should the AI step live?":

Path	Best for	Cost shape	Governance
Copilot Studio generative answers	RAG-style Q&A over curated sources	Per-message licensing	Within Copilot Studio's roof, includes content moderation
AI Builder Prompts	Reusable single-shot prompt within a flow or topic	AI Builder credits per call	Authored as a Dataverse component; visible in CoE Kit
Custom prompt-flow	Multi-step LLM workflow with branching, RAG, tool use	Compute + LLM tokens	Hosted in Azure ML; separate identity surface
Direct Azure OpenAI	Maximum control, custom RAG, fine-tuning, real-time streaming UX	Tokens directly; cheapest at scale	You own everything — moderation, abuse prevention, prompt logging

Default: stay inside Copilot Studio / AI Builder for governance. Only step out to prompt-flow / Azure OpenAI when there's a concrete requirement (cost at scale, custom RAG with vector store, fine-tuning, streaming UX) that justifies the operational overhead.

Premium connector / capacity note — every option here has cost. Call it out in the recommendation per §3 #8.

9. Regression test sets (the discipline)

Build a test set of 10-30 canonical prompts covering:

• Each topic's trigger-phrase range (2-3 prompts per topic).
• Each generative-answer scope (5-10 known questions with known correct answers).
• Adversarial / out-of-scope prompts (5+ — verify the bot redirects, doesn't hallucinate).
• Escalation triggers (verify escalation fires).
• Slot-filling validation (verify validation rejects bad input).

Re-run the entire set on every authoring change. Manually if necessary; automated via the Copilot Studio testing pane or Power Automate where possible. Save the test transcripts. A regression in test set = block the release.

10. Measurement (containment, escalation, resolution, CSAT)

• Containment rate — % of conversations the bot completes without escalating. Higher isn't always better; a bot that "contains" by hallucinating is worse than escalating.
• Escalation rate — % escalated. Track appropriate vs failure escalations.
• Resolution rate — for transactional bots, did the user actually complete the task?
• CSAT — end-of-conversation feedback. Two scales: was the answer accurate; was the experience good.

Watch trends, not absolutes. A drop after a release usually points to a knowledge source change or trigger phrase regression.

Anti-Patterns to Flag

• One mega-topic with 47 branches instead of decomposed topics
• "Set the bot loose on the intranet" knowledge configuration
• No regression test set ("we'll test it manually each release")
• No escalation path ("the bot will figure it out")
• System prompt copy-pasted from a tutorial, doesn't match the org voice or scope
• Slot-filling without confirmation before irreversible actions
• Trigger phrases that are all variants of one phrasing
• Hard-coded URLs, IDs, or secrets in topic responses (§3 #2 — env vars)
• Premium AI Builder / generative-answer usage without explicit licensing call-out (§3 #8)

Escalation

• Bot will handle PII / PHI / regulated content → ravenclaude-core security-reviewer on the data flow before any deployment.
• Bot calls flows that touch transactional systems → flow-engineer + power-automate skill for resilient action wiring; dataverse-architect if it writes to Dataverse.
• Need to ship via solution ALM → solution-alm-engineer + alm-pipeline-design skill. Bots are solution-aware; treat as code.
• Tenant DLP blocks the bot's required connector → power-platform-admin + dlp-policy-design skill, not a hallway exemption.
• Stuck on which AI path → architect in ravenclaude-core for the cost / governance / data-residency conversation.

When in doubt, ship narrower scope with stronger guardrails than wider scope with looser guardrails. A bot that handles 5 things well is a product. A bot that handles 500 things badly is a liability.