What Nobody Tells You About Building Production Voice AI Agents

The hard part isn't the LLM. It's making five real-time systems cooperate inside a one-second window — without ever letting the caller notice.

Building a voice AI agent sounds straightforward. You pipe audio into a speech model, feed the text to an LLM, convert the response back to audio, and play it to the caller. Simple, right?

It is — until you're in production and a caller is waiting. Then 200ms starts feeling like an eternity, a single transcription error cascades through your entire context window, and a TTS hiccup sounds like a frozen phone line. The LLM was always the easy part. The hard part is systems integration under real-time constraints.

This post covers the architectural lessons we've accumulated building Rymi, a Voice AI infrastructure platform. None of this is theoretical — it's what actually breaks in production.

The architecture, plainly

Every production voice agent is the same five-stage pipeline:

Phone / SIP → STT → Context + LLM → TTS → Audio out

The problem isn't any individual component. The problem is that each of them must fit inside roughly one second total — the threshold after which a human caller notices a delay and starts doubting whether the system is working.

The 1-second budget

Before writing a single line of code, budget your latency like a strict expense account:

Component	Target budget
SIP ingestion + audio buffer	~200ms (fixed cost, unavoidable)
Speech-to-text	300–400ms
LLM inference	200–300ms
Text-to-speech	200–300ms
Network round-trips	100–200ms
Retry / overhead buffer	100–200ms

Miss any one of these targets and the entire experience degrades. There's no single component to optimize — you're constantly balancing the whole system.

Speech-to-text: speed beats accuracy

This is counter-intuitive. You'd assume you want the most accurate transcription possible. In practice, a 97%-accurate STT that takes 800ms feels worse than a 93%-accurate STT that takes 300ms. Users perceive responsiveness, not accuracy.

Provider	Latency	Accuracy	Cost
Deepgram Nova-2 (recommended)	200–400ms	95%+	$0.0043/min
Google Cloud Speech	500–800ms	95%+	$0.006/min
Azure Cognitive Services	400–600ms	95%+	$0.005/min
Whisper (self-hosted)	800–1500ms	94%+	Compute only

Provider figures as of April 2026, from each vendor's published documentation. Latency and pricing move; check the vendor before you commit.

For phone calls, Deepgram with streaming is currently the strongest option. It supports interim_results and VAD (Voice Activity Detection) events, which means you can show the caller an acknowledgment signal while waiting for the final transcript — this alone shaves 200ms off perceived latency.

Phone audio is not regular audio. Phone calls are typically 8kHz narrowband, compressed, and acoustically degraded. Don't use a generic STT model trained on clear speech. Deepgram's "phone" model is specifically trained for this. The accuracy delta is significant.

LLM inference: stream, don't batch

Standard LLM API calls have 500ms+ latency before you receive the first token. If you wait for a complete response before starting TTS, you've already consumed more than half your budget on one component.

The solution is streaming at every stage — start TTS generation the moment you have the first complete sentence from the LLM, not when the full response is done:

// Start TTS as soon as a sentence boundary arrives
for await (const chunk of llmStream) {
  buffer += chunk;

  // Detect sentence boundary
  if (/[.!?]\s*$/.test(buffer)) {
    ttsQueue.push(buffer);
    buffer = '';
    if (!ttsPlaying) playNextChunk();
  }
}

The audio for the first sentence is already playing by the time the LLM finishes generating the second. The caller experiences a continuous response, not a loading pause.

Model selection

Model	Latency	Cost per 1M tokens	Best for
Claude 3 Haiku (sweet spot)	300–500ms	$0.25 in / $1.25 out	Speed + quality balance
GPT-4o Mini	300–500ms	~$0.15 in / $0.60 out	Cost-sensitive workloads
GPT-4 Turbo	600–1000ms	$10 in / $30 out	Complex reasoning tasks
Mistral 7B (self-hosted)	50–200ms	Compute only	Privacy-critical / offline

Model figures as of April 2026. Newer tiers have since shipped — see the model picker for what Rymi runs today.

For voice agents handling qualification, booking, or FAQ tasks, Claude 3 Haiku consistently fits the latency budget while producing natural-sounding responses. Save the heavier models for post-call analysis, not live calls.

Context management: the part most teams under-build

Raw transcribed text has no context. "That works for me" means nothing without knowing what was just discussed. You need a thin context layer that tracks conversation state, extracts entities (budget, name, phone number), and builds the prompt for each LLM call.

A mid-tier LLM with good conversation context will outperform a frontier model with no context. Build context management before you worry about model selection.

The design doesn't need to be complex. A simple class that tracks history, entities, and state — and builds a system prompt from them — handles the vast majority of real-world voice agent scenarios. Resist the urge to build a full NLU pipeline. The LLM's context window does most of that work for you.

Text-to-speech: queue, don't wait

TTS at high quality takes 500–1000ms for a full sentence. If you wait for the complete synthesis before playing audio, you're compounding every other latency. The pattern to use:

Send the first sentence to TTS as the LLM generates it
Queue subsequent sentences so they're ready as the first finishes playing
Never wait for the full TTS response — stream audio chunks as they arrive

ElevenLabs Turbo v2.5 (500–1000ms, high quality) and Google Cloud TTS (1–2s, good quality) are the two most common choices. OpenAI's TTS sits in between at $0.015/1k characters. The difference between providers matters less than whether you're streaming the output or waiting for it.

Failure handling is not optional

Real-time systems fail. Networks drop. STT misrecognizes. LLM calls time out. The difference between a professional voice agent and a frustrating one is whether you've planned for these scenarios before they happen.

The key insight is that you have less than 100ms to decide what to do when something fails. That decision needs to be pre-planned, not reasoned about in the moment:

async function processInput(audio) {
  try {
    const text = await timeout(stt.transcribe(audio), 400);
    const reply = await timeout(llm.generate(context, text), 300);
    return await timeout(tts.synthesize(reply), 300);
  } catch (err) {
    if (err.component === 'stt') return tts.synthesize("Sorry, I didn't catch that.");
    if (err.component === 'llm') return tts.synthesize("Let me connect you with someone.");
    if (err.component === 'tts') return sendSMS(storedReply); // fallback channel
  }
}

Five lessons from production

Latency matters more than accuracy. Users judge how fast you respond, not how correct you were. Optimize for latency first. Accuracy improvements that cost latency are usually the wrong trade.
Stream everything. Streaming every component — STT interim results, LLM tokens, TTS audio chunks — reduces perceived latency by 200–500ms without changing actual backend performance.
Know your fallbacks before you need them. You have no time to think when a component fails. Map every failure mode to a scripted response before you ship anything to production.
Phone audio is a different medium. 8kHz, narrowband, compressed. Models trained on clean audio behave differently. Use phone-specific STT models and test with real carrier audio, not clean microphone input.
You can't fix what you can't measure. Track component-level latencies, STT accuracy, LLM error rate, and outcome metrics (booking rate, transfer rate) from day one. Aggregate latency alone won't tell you where to optimize.

Closing thoughts

Building production voice agents is a systems engineering problem, not an AI research problem. The LLM is genuinely the easy part now. The hard work is making five real-time services cooperate inside a one-second window, gracefully handling failure in every one of them, and maintaining enough observability to know when something is quietly degrading.

Start simple: Deepgram for STT, Claude Haiku or GPT-4o Mini for the LLM, ElevenLabs for TTS. Measure everything from the first call. Optimize latency before accuracy. Build fallbacks before you think you need them.

The goal is a voice agent that is fast, reliable, and forgettable — in the best possible way. Callers should hang up having accomplished what they called for, with no memory of anything going wrong.

If you're building voice agents at scale and want to skip the infrastructure work, Rymi provides a full voice stack via REST API — STT, LLM, TTS, barge-in, and post-call transcripts, all handled for you.