Talk to most voice AI and you’ll feel it before you can name it: the pause. This is the architecture we use to remove it — two language models running in parallel, one fast enough to feel human, one smart enough to be right.

Pattern: two parallel LLMs · Fast lane: ~700 ms pre-response budget · Fallback: static localized filler · Invariant: never dead air

---

200–500 ms	~700 ms	2	0
natural human response gap	fast-lane budget	models per turn	ms of dead air allowed

---

01 · The awkward pause

You finish your sentence. Silence. A beat. Another beat. Then the agent starts talking. It’s less than a second of dead air, but your brain reads it instantly as this thing isn’t really listening — the same discomfort you feel on a bad phone line.

Here’s the counterintuitive part: that pause usually has nothing to do with how smart the model is. You can bolt on the most capable reasoning model in the world and the conversation still feels broken, because the thing humans actually judge in a voice interaction is timing, not IQ. Research on conversational turn-taking puts the natural human response gap at somewhere around 200–500 milliseconds. Cross that comfortably and the agent feels alive. Blow past it and no amount of eloquence saves you.

So the central engineering problem of voice AI isn’t “make the model smarter.” It’s “make a smart model feel instant.” This post is about the architecture we use to do exactly that — a two-parallel-LLM design where a fast model and a smart model run at the same time — and, along the way, a high-level tour of how a voice agent is actually assembled.

We’ll start gentle and ramp into the technical detail, so whether you’re deciding whether to build one or actually building one, there’s something here for you.

---

02 · Anatomy of a voice agent

Before the clever part, the foundation. A real-time voice agent is a loop — audio comes in, meaning is extracted, a response is generated, speech goes out — and every stage in that loop is streaming, because waiting for any stage to finish before starting the next one is how you get the awkward pause.

Walking through the stages:

1 · Transport / session layer. The user’s microphone and speaker join a real-time “room” — typically over WebRTC (for browsers and phones) or a raw WebSocket. This layer handles the messy realities of live audio: jitter, packet loss, echo cancellation, and getting sub-100 ms media latency across a network. Each session also carries its own configuration — who the agent is, which voice it uses, what tools it can call. In our system that config travels as a signed token: the client presents a JWT that encodes the tenant, the session, and the runtime settings, so the agent worker knows exactly which persona to spin up the moment the call connects.

2 · Speech-to-text (STT). Incoming audio is transcribed continuously. Good STT is streaming: it emits partial transcripts as you speak (“what’s the…”, “what’s the weather…”) and a final transcript when you pause. Those partials aren’t just for show — as we’ll see, they let us start working before you’ve even finished talking.

3 · Turn detection. This is the unsung hero. The agent needs to know when you’ve actually stopped talking versus when you’ve merely paused to think. Get it wrong in one direction and the agent talks over you; wrong in the other and it waits forever. This stage combines voice activity detection (VAD), silence timing (endpointing), and increasingly a small model that predicts, from your words and prosody, whether your turn is genuinely complete. A related problem lives here too: barge-in — letting you interrupt the agent mid-sentence. We use personalized VAD that locks onto the target speaker’s voice, so a TV in the background or a colleague across the room doesn’t falsely trigger an interruption. (In noisy rooms, naive VAD false-interrupts roughly a third of the time; speaker-targeted VAD cuts that dramatically.) We also filter out coughs and non-speech noise so they never get mistaken for a turn.

4 · The LLM — the brain. Once turn detection says “go,” the transcript plus conversation history goes to a language model, which streams back a response token by token. This is also where tools live: the model can call functions to look things up, hit an API, query a knowledge base, or hand off to another agent. Tool calls are where a lot of latency hides, because now you’re waiting on the model and a network round-trip to some external system.

5 · Text-to-speech (TTS). The model’s text stream is converted to audio and played back. The metric that matters here is time-to-first-audio — how fast the first chunk of speech comes out — because that’s what closes the gap the user is perceiving. Good TTS starts speaking the beginning of a sentence while the end of it is still being generated.

6 · The orchestrator. Tying all of this together is a worker process that manages the session lifecycle, wires the stages into each other, streams data between them, tracks conversation state, handles interruptions, and records metrics. It’s the conductor. Everything below is a decision the orchestrator makes.

---

03 · Why one LLM isn’t enough

The obvious knob to turn is the LLM, and here you hit a wall that no single model gets past.

You face a genuine tradeoff:

• A big, smart model reasons well, uses tools correctly, stays on-brand, handles the weird edge cases. But it’s slow — and if it needs to make a tool call before answering, it’s really slow, because now the clock includes an external round-trip.
• A small, fast model responds in a blink. But it’s shallower: worse at multi-step reasoning, less reliable with tools, more likely to say something you’ll regret.

Pick the smart model and the conversation drags. Pick the fast model and the conversation is snappy but dumb. Turn one knob and the other gets worse. This is the core dilemma of conversational AI, and you cannot resolve it by choosing better — the two things you want are in tension within a single model.

Unless you stop thinking of it as one model.

---

04 · The two-parallel-LLM architecture

Here’s the reframe. Humans don’t actually answer questions instantly either. When someone asks you something non-trivial, you don’t sit in silence computing the perfect answer and then deliver it whole. You say “Good question — let me think…” and you start reasoning out loud, buying yourself time while signalling that you’re engaged.

That’s the whole trick. We run two models in parallel: a fast one whose only job is to be immediate and natural, and a smart one whose job is to be right. They race, and the fast one’s job is to cover the gap while the smart one catches up.

Let’s break down what actually happens the instant turn detection fires:

Both lanes launch at once. The same context — transcript, history, persona — goes to both models simultaneously. No waiting.

The fast lane speaks first. The fast model is a small, low-latency LLM running under a tight budget — a hard ceiling of a couple of seconds, but really aiming to produce something inside a ~700 ms pre-response window. It runs at minimal reasoning effort (it isn’t trying to solve anything), and it often lives on a different, latency-optimized endpoint than the main model — the fast lane and the smart lane don’t even have to be the same provider or the same hardware. Its entire assignment is to emit one short, natural spoken opener — five to ten words — that a human would plausibly say while thinking: “Sure, let me pull that up for you.” The moment those words exist, TTS starts speaking them. The gap is closed.

The smart lane carries the truth. Meanwhile, the main reasoning model — full capability, tool access, no time pressure — is computing the actual answer. Crucially, the opener is designed to flow into whatever the main model says next. The fast model isn’t guessing the answer and risking being wrong; it’s producing connective tissue. “Let me pull that up for you… “ → ”…your order shipped Tuesday and arrives Thursday.“ To the user it’s one smooth, continuous utterance. They never hear a seam, and they never hear silence.

There’s always a floor. What if the fast model itself is slow — cold start, a hiccup, a bad moment? Then we fall back to a pre-written, localized filler phrase, chosen instantly with zero model call. We run this in a hybrid mode: use the fast LLM’s opener when it makes its budget, drop to a static phrase when it doesn’t. The invariant we protect above all else: there is never dead air. Something natural always comes out of the speaker inside the perceptible window, every single turn.

The same idea, everywhere

Once you internalize “do work in parallel to hide latency,” it stops being one feature and becomes a design philosophy that shows up all over the pipeline:

• Speculative tool prefetch. Remember those partial transcripts from STT? If we can already tell where a request is heading — a few words in, before you’ve even finished — we can start the tool call early. By the time you stop talking, the lookup may already be done. We’re spending a little wasted work on wrong guesses to buy latency on the right ones.
• Background consult and handoff. A primary agent can quietly consult a specialist in the background without transferring you, or fully hand off the call to a more specialized agent when the conversation calls for it. Either way, the coordination happens off the critical path so you’re not left listening to silence while the system reorganizes itself behind the scenes.

Different mechanisms, one thesis: latency is beaten by parallelism, not just by faster parts.

What it costs

No architecture is free, and it’s worth being honest about the bill:

• More inference. You’re running (at least) two models per turn instead of one. The fast lane is cheap and small, so it’s not double the cost — but it isn’t zero either.
• Reconciliation is fiddly. Making the opener flow seamlessly into the main answer takes real prompt engineering and careful stream handling. A clumsy handoff produces an audible seam — “Let me check… um, so, your order…” — which is its own kind of broken.
• Interruptions get harder. With speech starting faster and two lanes in flight, barge-in handling has to be rock-solid — which is exactly why the speaker-targeted VAD from §02 matters so much. You’ve made the agent eager to talk; now it had better be equally good at shutting up the instant the user does.

For a voice product, we think this trade is overwhelmingly worth it. Compute is cheap and getting cheaper. The awkward pause is expensive and never stops costing you.

---

05 · How we actually build one

Zooming back out, here’s the high-level recipe for standing up a voice agent with this architecture. Think of it as assembling the loop, then layering the parallelism on top.

---

06 · Lessons from production

A few things we learned the expensive way — all of which will bite anyone building in this space, regardless of the specific tools:

• Authenticated streaming connections fail silently. When your STT/TTS models sit behind authenticated WebSockets and the auth header is missing or wrong, you don’t always get a clean error — sometimes the socket opens and just sits there, deaf. Wrap every connection in the same auth helper and never assume a stage is healthy just because it connected.
• Cold starts are a latency cliff. The first request to a scaled-to-zero model server can take seconds to warm up — enough to blow every budget in this post. Warm up models on startup, health-check aggressively, and let the fast-lane fallback cover you while a cold replica spins up.
• Guard time-to-first-audio above all. It’s tempting to optimize total response time, but the user is judging the first chunk of sound. Everything after that can stream. Protect that first chunk like it’s the whole product — because, perceptually, it is.
• Racing is also resilience. A pleasant side effect of running two lanes and keeping static fallbacks: when one model or provider goes unhealthy, the system degrades gracefully instead of going silent. The architecture you built for speed turns out to also buy you reliability — the fallback that hides a slow model also hides a broken one.

---

07 · The takeaway

The instinct, when a voice agent feels bad, is to reach for a smarter model. Almost always, the real fix is architectural: stop making the user wait for intelligence.

Two LLMs in parallel — one fast enough to feel human, one smart enough to be right — resolve a tradeoff that no single model can. And the principle generalizes well beyond voice: any time a system has to feel instant while doing something genuinely hard, the move is to answer immediately with something cheap and true-enough, then reconcile with the expensive, correct result behind it. Autocomplete does it. Optimistic UIs do it. Your own brain does it every time you say “good question.”

For conversational AI, the lesson is blunt: latency architecture is product quality. Get the timing right and the intelligence finally gets to shine. Get it wrong and nobody sticks around long enough to find out how smart your agent is.

---

Vendor-neutral on purpose — the pattern matters more than the parts. © 2026 Oshara AI engineering log. The model was never the slow part.