Retell AI Reference Architecture

Overview

Production architecture for AI voice agents with Retell AI. Covers agent design, LLM configuration, telephony integration, custom tool functions, and call analytics for conversational AI applications.

Prerequisites

• Retell AI account with API key
• retell-sdk npm package
• Twilio or phone number for inbound/outbound calls
• WebSocket server for custom LLM (optional)

Architecture Diagram

set -euo pipefail
┌──────────────────────────────────────────────────────┐
│              Phone/Web Interface                      │
│  Twilio Number │ Web Call │ SIP Trunk                 │
└──────────┬───────────────────────────────────────────┘
           │
           ▼
┌──────────────────────────────────────────────────────┐
│              Retell AI Platform                       │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────┐   │
│  │ Voice Agent  │  │ LLM Engine   │  │ Voice     │   │
│  │ (config)     │  │ (prompts)    │  │ (TTS/STT) │   │
│  └──────┬───────┘  └──────┬───────┘  └───────────┘   │
│         │                 │                           │
│         ▼                 ▼                           │
│  ┌──────────────────────────────────────────────┐     │
│  │         Custom Tool Functions                 │     │
│  │  Book Appt │ Check Status │ Transfer Call    │     │
│  └──────────────────────┬───────────────────────┘     │
└─────────────────────────┼───────────────────────────┘
                          │
                          ▼
┌──────────────────────────────────────────────────────┐
│              Your Backend                             │
│  Webhook Handler │ Tool Endpoints │ Call Analytics    │
└──────────────────────────────────────────────────────┘

Instructions

Step 1: Agent and LLM Configuration

import Retell from 'retell-sdk';

const retell = new Retell({ apiKey: process.env.RETELL_API_KEY! });

// Create LLM with personality and tools
async function createAgentLLM() {
  const llm = await retell.llm.create({
    model: 'gpt-4o-mini',
    general_prompt: `You are Sarah, a friendly appointment scheduling assistant.

Instructions:
- Greet callers warmly and ask how you can help
- Collect: name, preferred date/time, reason for appointment
- Confirm all details before booking
- Keep responses under 2 sentences
- Be conversational, not robotic`,
    begin_message: "Hi there! This is Sarah. How can I help you today?",
    general_tools: [
      {
        type: 'end_call',
        name: 'end_call',
        description: 'End call when conversation is complete',
      },
      {
        type: 'custom',
        name: 'book_appointment',
        description: 'Book an appointment after collecting all details',
        url: `${process.env.BASE_URL}/api/retell/book`,
        parameters: {
          type: 'object',
          properties: {
            name: { type: 'string', description: 'Caller name' },
            date: { type: 'string', description: 'Preferred date' },
            time: { type: 'string', description: 'Preferred time' },
            reason: { type: 'string', description: 'Appointment reason' },
          },
          required: ['name', 'date', 'time'],
        },
      },
    ],
  });

  return llm;
}

Step 2: Voice Agent Setup

async function createVoiceAgent(llmId: string) {
  return retell.agent.create({
    agent_name: 'Appointment Scheduler',
    response_engine: { type: 'retell-llm', llm_id: llmId },
    voice_id: 'eleven_labs_rachel',
    language: 'en-US',
    responsiveness: 0.8,
    interruption_sensitivity: 0.7,
    enable_backchannel: true,
    voice_speed: 1.0,
    voice_temperature: 0.5,
  });
}

Step 3: Tool Function Endpoints

import express from 'express';
const app = express();

// Tool function: book appointment
app.post('/api/retell/book', express.json(), async (req, res) => {
  const { name, date, time, reason } = req.body.args;

  const booking = await bookAppointment({ name, date, time, reason });

  res.json({
    result: `Appointment booked for ${name} on ${date} at ${time}. Confirmation number: ${booking.id}`,
  });
});

// Webhook: call events
app.post('/api/retell/webhook', express.json(), async (req, res) => {
  const { event, call } = req.body;

  if (event === 'call_ended') {
    await saveCallRecord({
      callId: call.call_id,
      duration: call.end_timestamp - call.start_timestamp,
      transcript: call.transcript,
      sentiment: call.call_analysis?.sentiment,
    });
  }

  res.json({ received: true });
});

Step 4: Initiate Outbound Calls

async function makeOutboundCall(
  toNumber: string,
  agentId: string,
  metadata?: Record<string, string>
) {
  return retell.call.createPhoneCall({
    from_number: process.env.RETELL_PHONE_NUMBER!,
    to_number: toNumber,
    override_agent_id: agentId,
    metadata,
  });
}

Error Handling

Issue	Cause	Solution
Agent not responding	LLM prompt too complex	Simplify prompt, use shorter responses
Tool call fails	Webhook URL unreachable	Verify endpoint is publicly accessible
Poor voice quality	High latency	Use gpt-4o-mini, increase responsiveness
Call drops	WebSocket timeout	Check network stability, add reconnection

Examples

Quick Agent Test

// Create a web call for testing (no phone needed)
const webCall = await retell.call.createWebCall({
  agent_id: agentId,
  metadata: { test: 'true' },
});
console.log(`Test call URL: ${webCall.call_id}`);

Resources

• Retell AI Documentation
• Retell Agent Setup
• Retell API Reference

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale