Build a Voice IVR With Twilio + GPT-5 + ElevenLabs: A 1-Day Tutorial (and the 3 Gotchas)

Most automated phone lines feel like a 2009 IVR: "press 1 for sales." With Twilio Media Streams, GPT-5, and ElevenLabs, you can build one that a caller can just talk to — interrupt mid-sentence, ask in their own words, get a natural-sounding reply in about a second. We built a working version in a day. The wiring is straightforward; what wrecks naive builds is three gotchas around streaming, interruption, and latency. Here's the full build and exactly how to avoid them.

The 60-Word Answer

Twilio Media Streams sends the live call audio to your server over a WebSocket. You stream that audio to a speech-to-text engine, feed the transcript to GPT-5, stream GPT-5's reply through ElevenLabs to get natural speech, and send that audio back to Twilio. The three gotchas: stream everything (don't wait for full turns), implement barge-in (stop talking when the caller speaks), and budget every millisecond of latency.

Why Streaming, Not Twilio's Old <Gather>

Twilio's classic <Gather> verb waits for the caller to finish, transcribes, then responds — a walkie-talkie, not a conversation. Media Streams sends raw audio to your server in real time over a WebSocket, so you can transcribe as the caller speaks and start replying the instant they pause. That's the difference between "this feels like a robot" and "this feels like a person." GPT-5's faster responses and ElevenLabs' streaming TTS make the round trip fast enough that the caller doesn't notice the machinery.

What You'll Need

Step 1: Point Twilio at Your WebSocket

When a call comes in, Twilio fetches TwiML. You return a <Connect><Stream> that opens a WebSocket from Twilio to your server.

// server.js — the inbound-call webhook
  app.post("/incoming", (req, res) => {
    res.type("text/xml").send(
      <Response>
        <Connect>
          <Stream url="wss://your-server.example.com/media" />
        </Connect>
      </Response>
    );
  });

Twilio now opens a WebSocket to /media and starts sending audio frames as base64-encoded μ-law at 8kHz. Every frame carries a streamSid you'll need to send audio back.

Step 2: The Media WebSocket — Stream Audio Both Ways

This is the heart of it. You receive caller audio, push it to STT, and when you have reply audio, send it back to Twilio on the same socket.

// media.js
  import { WebSocketServer } from "ws";
  const wss = new WebSocketServer({ noServer: true });
  wss.on("connection", (ws) => {
    let streamSid = null;
    const stt = openStreamingSTT();      // your STT WebSocket
    const convo = newConversation();      // GPT-5 message history
    ws.on("message", (raw) => {
      const msg = JSON.parse(raw);
      if (msg.event === "start") {
        streamSid = msg.start.streamSid;
      }
      if (msg.event === "media") {
        // caller audio chunk → push to STT
        stt.sendAudio(Buffer.from(msg.media.payload, "base64"));
      }
      if (msg.event === "mark") {
        // Twilio finished playing a chunk we sent (used for barge-in)
      }
    });
    // When STT emits a final transcript, run GPT-5 + TTS and stream back
    stt.onFinal(async (transcript) => {
      const replyAudioChunks = await think_and_speak(convo, transcript);
      for (const chunk of replyAudioChunks) {
        ws.send(JSON.stringify({
          event: "media",
          streamSid,
          media: { payload: chunk.toString("base64") }
        }));
      }
      // a mark lets us know when playback finishes
      ws.send(JSON.stringify({ event: "mark", streamSid,
                               mark: { name: "reply-done" } }));
    });
  });

The audio you send back must also be 8kHz μ-law to match Twilio. ElevenLabs can output the right format, or you resample. Get this wrong and the caller hears chipmunk audio or static — the most common first-build symptom.

Step 3: Think and Speak (GPT-5 → ElevenLabs, Streamed)

To hit ~1 second, you stream GPT-5's text into ElevenLabs as it generates, so speech synthesis starts before GPT-5 finishes its sentence.

// think_and_speak.js
  import OpenAI from "openai";
  const openai = new OpenAI();
  export async function think_and_speak(convo, transcript) {
    convo.push({ role: "user", content: transcript });
    // 1. Stream GPT-5 tokens
    const stream = await openai.chat.completions.create({
      model: "gpt-5-mini",            // mini is plenty for IVR; upgrade if needed
      messages: convo,
      stream: true,
      temperature: 0.4
    });
    const audioChunks = [];
    let sentence = "";
    for await (const part of stream) {
      const delta = part.choices[0]?.delta?.content || "";
      sentence += delta;
      // 2. When we have a full clause, send it to ElevenLabs immediately
      if (/[.!?]/.test(delta)) {
        const audio = await elevenLabsTTS(sentence);  // 8kHz μ-law out
        audioChunks.push(audio);
        sentence = "";
      }
    }
    if (sentence.trim()) audioChunks.push(await elevenLabsTTS(sentence));
    return audioChunks;
  }

Synthesising per-clause instead of per-full-reply is the trick that cuts perceived latency: the caller hears the first part of the answer while the rest is still being generated.

The 3 Gotchas That Wreck Naive Builds

Gotcha 1: Audio format mismatch (μ-law, 8kHz)

Gotcha 2: No barge-in (the bot talks over the caller)

// barge-in: caller spoke while bot was talking → stop playback
  stt.onSpeechStarted(() => {
    if (botIsSpeaking) {
      ws.send(JSON.stringify({ event: "clear", streamSid }));
      botIsSpeaking = false;
    }
  });

Gotcha 3: Latency stack-up

When NOT to Build This

Real Example: A Restaurant Reservation Line

We built the Twilio + GPT-5-mini + ElevenLabs pipeline above, with a Sarvam STT front end for the Hinglish callers and a create_reservation function that wrote to their booking system. Barge-in was the feature diners noticed — they could cut off the greeting and say "table for four, 8pm, Indiranagar" and the bot just handled it. After-hours bookings stopped going to voicemail. We tuned the latency budget over the first week to keep first-sound under 1.2 seconds, which is where it stopped feeling like a machine.

The same streaming-and-barge-in architecture powers the conversation engine in TalkDrill, our in-house English-speaking app with 5,000+ users, where we cut voice round-trip latency to 740ms on Indian 4G. We've also shipped this for a Tally helpdesk voice IVR. Our AI automation team treats latency budgeting as the core engineering work, not a tuning afterthought.

Frequently Asked Questions

What audio format does Twilio Media Streams use?

Twilio sends and expects 8kHz μ-law (G.711) audio, base64-encoded, over the WebSocket. Your STT and TTS almost certainly use a different sample rate, so you must resample on both edges. Mismatched formats are the single most common reason a first build produces static or chipmunk speech.

How do I make a voice bot stop talking when the caller interrupts?

Implement barge-in: when your STT detects the caller has started speaking while the bot is still playing audio, send Twilio a clear message to halt playback, then process the new input. Without this, the bot talks over callers and the experience feels broken.

What latency should I target for a voice IVR?

Aim for about 1–1.2 seconds from when the caller stops speaking to first audio out. Get there by overlapping the pipeline — use partial transcripts, stream the model's tokens, and synthesise speech per clause rather than waiting for the full reply. Measure each hop to find the slow one.

Should I use GPT-5 or a smaller model for a voice IVR?

gpt-5-mini is usually plenty for an IVR — order status, bookings, FAQs are not deep-reasoning tasks, and the smaller model is cheaper and faster, which helps your latency budget. Reserve the flagship GPT-5 for genuinely complex conversations where the mini model picks wrong actions.

Can the voice bot handle Hindi and Hinglish callers?

Yes. Put a code-switch-aware STT like Sarvam in front for Indian-language callers, and instruct the model to reply in the caller's language. ElevenLabs and other TTS engines support Hindi voices. The streaming and barge-in architecture is identical regardless of language.

Do I need a server, or can this run serverless?

You need a persistent WebSocket connection for the duration of each call, so a long-lived server (or a platform that supports WebSockets) works best. Short-lived serverless functions struggle with the open socket. For local development, ngrok exposes your laptop; for production, run a real Node server.

As Hrishikesh, our CTO, says: callers forgive a wrong answer faster than they forgive being talked over. Build barge-in first, polish the brain second.