azure-openai

Azure OpenAI

When to Use

• Deploying and integrating GPT-4o, GPT-4, GPT-3.5-turbo, or embedding models on Azure
• Building RAG (Retrieval-Augmented Generation) pipelines with Azure AI Search
• Designing prompt engineering patterns (system messages, few-shot, chain-of-thought)
• Implementing Responsible AI with Content Safety filtering
• Choosing between standard and provisioned deployment for throughput requirements
• Preparing for Azure AI Engineer Associate (AI-102) or AI-200 exam

Core Jobs

1. Model Deployment

Model Family	Use Case
GPT-4o	Latest multimodal; text + vision; balanced cost/performance
GPT-4	High-quality reasoning; complex tasks
GPT-3.5-turbo	Fast, cost-effective; simple tasks, high volume
text-embedding-ada-002 / text-embedding-3-*	Vector embeddings for semantic search
DALL-E 3	Image generation from text
Whisper	Speech-to-text transcription

• Deployment types:

  Type	Throughput	Billing	Best For
  Standard	Shared; TPM/RPM limits	Pay-per-token	Variable workloads
  Provisioned (PTU)	Reserved capacity	Hourly PTU rate	Predictable, high-throughput
  Global Standard	Globally routed	Pay-per-token	Overflow capacity

• TPM = Tokens Per Minute; RPM = Requests Per Minute; quotas per deployment per region

2. RAG (Retrieval-Augmented Generation)

• Pattern: User query → retrieve relevant chunks → pass as context to LLM → generate grounded response
• Azure implementation:
  1. Document ingestion: chunk documents → generate embeddings → index in Azure AI Search
  2. Retrieval: embed user query → search AI Search (vector, keyword, or hybrid) → return top-k chunks
  3. Generation: send chunks + query to Azure OpenAI → GPT generates answer citing retrieved context
• Azure AI Search capabilities:
  • Vector search: cosine similarity on embedding vectors
  • Keyword search: BM25 full-text search
  • Hybrid search: combine vector + keyword scores (recommended)
  • Semantic ranking: rerank hybrid results using language understanding
• Chunking strategy: fixed-size (tokens), sentence-based, or semantic chunking; overlap recommended

3. Prompt Engineering Patterns

Pattern	Description	When to Use
System message	Persona, instructions, constraints	Always; sets model behavior
Few-shot examples	Input-output examples in prompt	Specific output format required
Chain-of-thought	"Think step by step" instruction	Complex reasoning tasks
JSON mode	response_format: {type: "json_object"}	Structured output for parsing
Tool/function calling	Define functions; model decides to call	Agent-style applications

• System message persists across conversation turns; user/assistant messages are conversation history
• Temperature: 0 = deterministic; 1+ = creative; use 0 for RAG grounding, 0.7 for creative tasks
• Max tokens: limit output length; prevent runaway token consumption

4. Content Safety

• Azure AI Content Safety = content moderation for inputs and outputs
• Content filters applied by default; categories: Hate, Violence, Sexual, Self-Harm
• Severity levels: Safe, Low, Medium, High; configurable threshold per category (requires approval)
• Custom blocklist: add domain-specific prohibited terms
• Groundedness detection: detect hallucinations (responses not grounded in retrieved context)
• Prompt Shield: detect jailbreak attempts and indirect prompt injection
• All content safety events logged; reviewable in Azure AI Studio

5. Function Calling / Tools

• Define JSON schema of available functions; model includes function call in response when appropriate
• Client-side execution: parse model's function call → execute → return result → model generates final answer
• Use cases: search the web, query database, get current time, call external API
• Parallel function calling: model can call multiple functions in one response (GPT-4o, GPT-4-turbo)
• tool_choice: "auto" = model decides; "required" = must call a tool; specific tool name = force specific tool

6. Fine-Tuning

Aspect	Detail
Available models	GPT-4o mini, GPT-3.5-turbo
Data format	JSONL with {messages: [{role, content}]} per example
Min training examples	10 (recommended: 50-100+)
Use case	Domain-specific tone/format; not for adding new knowledge

• Fine-tuning ≠ RAG: fine-tuning adjusts model behavior/style; RAG provides retrieval-time knowledge
• Rule of thumb: try prompt engineering + few-shot first; fine-tune only when prompt alone is insufficient
• Fine-tuned model deployed like standard model; billed per token + training compute

Key Concepts

• PTU (Provisioned Throughput Units) — reserved capacity; predictable latency; no rate limits during reservation
• RAG — retrieval at inference time; grounds model in current data; no retraining needed
• Vector search — cosine similarity on embeddings; finds semantically similar content
• Hybrid search — vector + keyword combined; better retrieval quality than either alone
• System message — model persona and instructions; persists across all turns in a conversation
• Content filter — default content safety filters; configure severity thresholds per category
• JSON mode — force model to output valid JSON; always include JSON instruction in system/user message

Checklist

• Azure OpenAI (not direct OpenAI API) used for enterprise compliance and private networking?
• PTU deployment evaluated for predictable high-throughput workloads?
• RAG implemented with Azure AI Search hybrid search + semantic ranking?
• System message crafted with clear persona, instructions, and constraints?
• Content Safety filters reviewed and custom blocklist configured for domain?
• JSON mode used for structured output instead of prompt-only JSON instructions?
• Private endpoint configured on Azure OpenAI resource for VNet isolation?

Output Format

• 🔴 Critical — Azure OpenAI public endpoint used for sensitive data without Private Endpoint
• 🔴 Critical — no content safety filtering on user-facing application (default filters disabled)
• 🟡 Warning — vector-only search without hybrid search (misses keyword-matched exact terminology)
• 🟡 Warning — temperature set to 1+ for RAG application (encourages hallucination; use 0 for grounding)
• 🟢 Suggestion — enable semantic ranking on Azure AI Search for best RAG retrieval quality

Exam Tips

• Azure OpenAI ≠ OpenAI API — Azure = enterprise features: private endpoints, RBAC, Entra ID auth, compliance certifications
• PTU (Provisioned Throughput Units) = reserved capacity — predictable latency; no rate limit surprises; billed hourly regardless of usage
• RAG = retrieval + generation — Azure AI Search provides retrieval (vector/hybrid); OpenAI generates grounded answer; no model retraining
• Vector search = semantic similarity; keyword = exact match; hybrid = both — hybrid recommended for RAG; semantic ranking reranks hybrid results
• System message = persona + instructions; persists across turns — always set for production apps; defines model's behavior throughout conversation
• Responsible AI: content filters always on by default — configure severity thresholds per category (requires Microsoft approval for lower thresholds)