shiny-aiconversation

Shiny.AiConversation Skill

You are an expert in the Shiny.AiConversation library, a centralized AI service for .NET MAUI applications that integrates chat, speech recognition, wake word detection, text-to-speech, and persistent message storage.

Library Overview

GitHub: https://github.com/shinyorg/speech (Shiny.AiConversation lives under the ai/ directory) NuGet: Shiny.AiConversation Namespace: Shiny.AiConversation Infrastructure Namespace: Shiny.AiConversation.Infrastructure (internal implementations)

The library provides:

• IAiConversationService: Central orchestrator for AI interactions — manages access checking, state (Idle/Listening/Thinking/Responding), wake word detection, speech-to-text capture, chat client communication, text-to-speech response, acknowledgement modes, sound effects, persistent chat history, conversation continuation (auto-listens when AI asks a question), and voice interruption (quiet words to stop TTS, or speak over the AI to redirect the conversation)
• IChatClientProvider: Abstraction for obtaining an IChatClient (from Microsoft.Extensions.AI) — a default implementation (InjectedChatClientProvider) resolves IChatClient from DI; custom implementations handle authentication, token management, and client construction
• IMessageStore: Abstraction for persisting and querying chat message history — implementations provide storage (SQLite, file system, cloud, etc.)
• ChatLookupAITool: AI tool that allows the AI to search past conversations via IMessageStore — automatically added by ContextProvider when an IMessageStore is registered
• VoiceSelectionContextProvider: Optional IContextProvider that adds three AI tools — get_available_voices, play_voice_sample, and change_voice — enabling the AI to list voices, play audio samples, and change its own TTS voice mid-conversation. Enabled via opts.AddVoiceSelectionTools().
• AiChatMessage: Record representing a persisted chat message with Id, Message, Timestamp, and Direction (User/AI)
• IContextProvider: Visitor-pattern abstraction for populating an AiContext per request. Each provider's Apply(AiContext) method receives a mutable context and adds its contributions. The ContextProvider handles time-based prompts, acknowledgement-aware voice prompts, and DI-registered AITool instances. Implement custom providers to add domain-specific system prompts, tools, or override speech settings.
• AiContext: Mutable context object passed to IContextProvider.Apply() — contains Acknowledgement, SystemPrompts, Tools, QuietWords, SpeechToTextOptions, and TextToSpeechOptions that providers populate or modify
• AiConversationOptions: Fluent configuration for DI registration — sets chat client provider, message store, sound provider, and additional context providers via AddContextProvider<T>()

Built-in Provider Packages:

• Shiny.AiConversation.OpenAi (OpenAiStaticChatProvider): Static OpenAI-compatible provider. Accepts API key, endpoint URI, and model name. Works with OpenAI, Azure OpenAI, Ollama, or any OpenAI-compatible API. Register with opts.AddStaticOpenAIChatClient(apiToken, endpointUri, modelName).
• Shiny.AiConversation.Maui.GithubCopilot (GitHubCopilotChatClientProvider): MAUI-specific provider using GitHub device code OAuth flow and the Copilot API. Self-contained auth — shows a popup with the device code, copies to clipboard, opens browser, polls until authorized. Tokens stored in SecureStorage. Register with opts.AddGithubCopilotChatClient(). Additional API: StartAuthentication(), CancelAuthentication(), SignOut(), IsAuthenticated, AccessTokenChanged event.

Dependencies

• Microsoft.Extensions.AI — IChatClient, ChatMessage, ChatRole, AITool, ChatOptions
• Shiny.Speech — ISpeechToTextService, ITextToSpeechService, IAudioPlayer for voice interactions and sound effects

When to Use This Skill

Invoke this skill when the user wants to:

• Set up an AI chat service in a .NET MAUI app
• Register and configure IAiConversationService with dependency injection
• Implement IChatClientProvider for a specific AI backend (OpenAI, GitHub Copilot, Azure, etc.)
• Implement IMessageStore for persistent chat history
• Add wake word detection to an app
• Configure acknowledgement modes (None, AudioBlip, LessWordy, Full)
• Set up sound effects for AI state transitions
• Configure voice interruption with quiet words
• Set up speech-to-text and text-to-speech options (culture, voice, speech rate, etc.)
• Add the optional ChatLookupAITool for AI-driven history search
• Enable voice selection tools so the AI can list voices, play samples, and switch its own voice
• Build a chat UI that integrates with IAiConversationService
• Handle AI state changes (Idle, Listening, Thinking, Responding)
• Use TalkTo or ListenAndTalk for AI interactions
• Check speech/microphone access before starting voice features

Code Generation Instructions

1. Registration (MauiProgram.cs)

Always register with AddShinyAiConversation():

using Shiny.AiConversation;

// Option A: Register an IChatClient in DI (simplest approach)
builder.Services.AddChatClient(new OpenAIClient("your-api-key").GetChatClient("gpt-4o").AsIChatClient());

// Option B: Use a built-in provider
builder.Services.AddShinyAiConversation(opts =>
{
    // OpenAI-compatible (OpenAI, Azure OpenAI, Ollama, etc.)
    opts.AddStaticOpenAIChatClient("your-api-key", "https://api.openai.com/v1", "gpt-4o");

    // OR GitHub Copilot (MAUI only — self-contained auth with SecureStorage)
    opts.AddGithubCopilotChatClient();

    opts.SetMessageStore<MyMessageStore>(); // optional — ChatLookupAITool added automatically
});

• Built-in providers: AddStaticOpenAIChatClient() for OpenAI-compatible APIs, AddGithubCopilotChatClient() for GitHub Copilot on MAUI
• SetChatClientProvider<T>() is for custom providers — if not set and no built-in provider is used, the default InjectedChatClientProvider resolves IChatClient from DI
• SetMessageStore<T>() is optional — enables persistent history; the ContextProvider automatically adds ChatLookupAITool when a store is present
• AddVoiceSelectionTools() is optional — registers VoiceSelectionContextProvider, giving the AI tools to list voices, play samples, and change its own voice
• AddContextProvider<T>() registers additional IContextProvider implementations
• SetSoundProvider<T>() registers a custom ISoundProvider implementation
• System prompts, tools, quiet words, and speech options are provided via IContextProvider implementations registered in DI (a ContextProvider is auto-registered)

2. Chat Client Setup

Simple approach — register IChatClient in DI (the default provider resolves it automatically):

builder.Services.AddChatClient(new OpenAIClient("your-api-key").GetChatClient("gpt-4o").AsIChatClient());

Advanced approach — implement IChatClientProvider for on-demand auth or token refresh:

public class MyChatClientProvider : IChatClientProvider
{
    public async Task<IChatClient> GetChatClient(CancellationToken cancelToken = default)
    {
        // Obtain/refresh tokens, authenticate if needed, build client
        return new OpenAIChatClient(...);
    }
}

• Handle token expiry and re-authentication inside GetChatClient
• Can inject INavigator to navigate to a login page if authentication is needed on-demand

3. Implementing IContextProvider (Custom)

The ContextProvider is registered automatically and provides time-based prompts, acknowledgement-aware voice prompts, and any AITool instances from DI. To add custom system prompts, tools, or modify speech settings, implement IContextProvider using the visitor pattern and register it:

public class MyContextProvider : IContextProvider
{
    public Task Apply(AiContext context)
    {
        context.SystemPrompts.Add("You are a helpful assistant for our company.");
        // context.Tools.Add(...) to add AI tools
        // context.Acknowledgement is available to adjust behavior per mode
        // context.QuietWords — modify quiet words for voice interruption
        // context.SpeechToTextOptions — set/override speech recognition options
        // context.TextToSpeechOptions — set/override text-to-speech options
        return Task.CompletedTask;
    }
}

// Register in DI — multiple providers are supported, executed in sequence
builder.Services.AddShinyAiConversation(opts =>
{
    opts.AddContextProvider<MyContextProvider>();
});

4. Implementing IMessageStore

public class MyMessageStore : IMessageStore
{
    public Task Store(string? userTriggeringMessage, ChatResponse response, CancellationToken cancellationToken) { ... }
    public Task Clear(DateTimeOffset? beforeDate = null) { ... }
    public Task<IReadOnlyList<AiChatMessage>> Query(
        string? messageContains = null,
        DateTimeOffset? fromDate = null,
        DateTimeOffset? toDate = null,
        int? limit = null,
        CancellationToken cancellationToken = default) { ... }
}

5. Using IAiConversationService

// Check access before using voice features
var access = await aiService.RequestAccess();
if (access != AccessState.Available)
{
    // Speech is not available — handle accordingly
    return;
}

// Send a text message
await aiService.TalkTo("What's the weather?", cancellationToken);

// Listen via microphone and send to AI
await aiService.ListenAndTalk(cancellationToken);

// Wake word detection (loops until stopped)
await aiService.StartWakeWord("Hey Assistant");
aiService.StopWakeWord();

// Query chat history
var history = await aiService.GetChatHistory(limit: 25);
var filtered = await aiService.GetChatHistory(messageContains: "weather", startDate: yesterday);

// Clear history
await aiService.ClearChatHistory();
await aiService.ClearChatHistory(beforeDate: oneWeekAgo);

// React to state changes
aiService.StatusChanged += (state) => { /* update UI with new AiState */ };
aiService.AiResponded += (response) =>
{
    // response.Response — the complete ChatResponse
    // response.Response.Text — the text content of the response
    // response.WasReadAloud — whether TTS was used
    // response.ExpectsResponse — true if AI asked a question (will auto-listen for reply)
};

6. Acknowledgement Modes

Mode	Behavior
None	No audio feedback or TTS
AudioBlip	Short sound effects at state transitions
LessWordy	TTS with "be concise" system prompt injected
Full	TTS with full unmodified responses

7. AI States

State	Description
Idle	Ready for input
Listening	Actively listening for speech
Thinking	Waiting for AI to process
Responding	AI is streaming its response

Best Practices

1. Use IContextProvider for configuration — System prompts, tools, quiet words, and speech options are all configured via IContextProvider.Apply(AiContext) — not set directly on the service
2. Handle auth on-demand — IChatClientProvider should handle authentication lazily, not force login at startup
3. Use TwoWay binding for Acknowledgement in settings UIs
4. Subscribe/unsubscribe to StatusChanged and AiResponded in page lifecycle (OnAppearing/OnDisappearing)
5. Use CancellationToken for all TalkTo/ListenAndTalk calls
6. MessageStore is optional — The service works without it but GetChatHistory/ClearChatHistory will throw
7. ChatLookupAITool is opt-in — Pass addAiLookupTool: true to SetMessageStore to allow the AI to search past conversations
8. No reflection — All registrations must be explicit; do not use ActivatorUtilities.CreateInstance or reflection-based patterns