Building Voice AI That Feels Human: A Latency Budget Breakdown
Sri Raghu Malireddi
Founder & CEO
Abhishake Kumar Bojja
Founding ML Engineer
The first sign that a voice agent is slow is usually not a latency graph. It is the user saying, "Hello?" a second time.
That second "Hello?" can create a surprisingly difficult failure loop. The agent may already be generating a response, but the user does not know that. They hear silence, assume the system did not hear them, and speak again. Speech recognition now has another utterance to reconcile, while the response that was already in flight may need to be interrupted. The user hears a clipped response, waits again, and may start changing how they speak to the system. Nothing is necessarily wrong with the underlying model; the problem is that the conversation has lost its timing.
This is one of the fundamental differences between Voice AI and chat applications. In a chat interface, latency can often be hidden behind visible progress. A typing indicator tells the user that the system is working, while streaming tokens allow them to start reading before the response is complete. Voice has no equivalent escape hatch. When the user finishes speaking, silence is the entire interface. They cannot see that a transcript is stabilizing, retrieval is running, or the model has started generating. If the response takes one or two seconds, the delay becomes ambiguous: did the agent hear me, is it still connected, or should I speak again?
That makes latency particularly important for Voice AI. A slow response does not simply make the application feel slower; it can interfere with turn-taking and cause the user and agent to speak over one another. The latency budget therefore has to be considered across the entire real-time AI stack, from endpointing and speech recognition through retrieval, inference, speech synthesis, transport, and playback. If one stage consumes more than its share, the effect is felt across the entire conversation.
Why Voice AI Has Less Room Than Chat
Chat applications have several ways to communicate that work is happening. They can display a typing indicator, stream partial text, or allow users to scan a response as it is generated. These cues turn waiting into visible progress and give the user confidence that the system is still working.
Voice AI does not have that benefit. Once the user yields the floor, the system has to communicate progress through its response. A period of silence provides no information about what is happening internally. This is why a delay that feels insignificant in a chat application can feel much more pronounced in a voice conversation.
The problem also becomes more serious as latency increases because the user can change their behavior. A user who waits one second may simply notice the delay. A user who waits two seconds may start wondering whether the system heard them. A user who waits several seconds may repeat the question. That second utterance can arrive while the first response is already being generated, forcing the system to detect the interruption, cancel work that is no longer relevant, and recover the conversational state.
In other words, Voice AI latency is not simply a measure of how quickly a system produces an answer. It affects the system's ability to correctly determine who has the floor.
The Latency Budget for a Single Voice AI Turn
For a voice agent, one of the most useful end-to-end metrics is time to first audio (TTFA):
TTFA = first_audio_played_at_user - end_of_user_speech
This measures the time between the user finishing their utterance and actually hearing the agent begin its response. That distinction matters because individual provider metrics do not necessarily represent the experience the user is having. A speech recognition provider returning a transcript quickly, an LLM producing its first token, or a text-to-speech service returning its first byte are all useful measurements for diagnosing the system, but none of them tells you when the user actually hears the response.
Our Production AI Stack uses approximately 800 milliseconds to first audio as a Voice AI design reference. This should be treated as a design target rather than a universal benchmark or customer SLA. The appropriate target will vary depending on the type of conversation, language, network conditions, device, and acceptable error rate. The important thing is to establish an explicit budget before making architectural decisions.
A representative turn can be broken down as follows:
| Stage | What must happen | Illustrative reference |
|---|---|---|
| Endpointing | Determine that the user has yielded the floor | Part of ~175 ms of shared residual headroom |
| Final speech transcript | Stabilize the words needed to route the turn | ~100 ms best case, ~200 ms typical |
| Retrieval | Fetch context for a grounded answer | ~250 ms best case, ~500 ms to 1.5 s typical |
| LLM | Produce the first useful tokens | ~200 ms best case, ~400 ms typical |
| Speech synthesis | Turn a stable phrase into the first audio | ~75 ms best case, ~200 ms typical |
| Transport and playback | Deliver, buffer, decode, and play audio | Part of ~175 ms of shared residual headroom |
| Full turn | User stops speaking to first audio played | 800 ms design reference |
The four quantified stages consume approximately 625 milliseconds in the best case. Against an 800-millisecond design reference, that leaves roughly 175 milliseconds for endpointing, transport, playback, and orchestration. In the typical ranges, those same four stages add up to approximately 1.3 to 2.3 seconds before separately measured endpointing and playback time are included.
This is why latency cannot be optimized one component at a time. A retrieval call that takes an additional 200 milliseconds, for example, may not look particularly concerning in isolation. But when that delay comes out of the same budget as inference, speech synthesis, and playback, it can be the difference between a response that feels immediate and one that causes the user to wonder whether the agent heard them.
Following a Voice AI Request Through the Stack
Consider a user asking, "Can I use this in a mobile app?" and then stopping speaking. The endpointing system first needs to determine that the user has actually finished the turn rather than simply pausing in the middle of a sentence. Speech recognition needs to stabilize the final transcript so that the application can correctly route the request. The application then retrieves the relevant product context, passes that context to the language model, and begins generating a grounded response. Once enough stable language is available, speech synthesis can begin producing audio, which then has to travel back to the device, be buffered and decoded, and ultimately be played through the speaker.
Some of these stages can overlap when doing so does not compromise correctness. For example, partial transcripts can be used to begin speculative retrieval, and stable portions of model output can be streamed directly into speech synthesis. However, the underlying dependencies remain. Retrieval needs a sufficiently stable query, generation needs the retrieved context, speech synthesis needs enough stable language to form a coherent phrase, and playback needs audio to have reached the client.
The user does not experience these as separate operations. They experience the sum of them as the amount of time between finishing a sentence and hearing a response.
That makes each stage worth examining individually.
1. Endpointing and Speech Recognition: Buy Certainty Deliberately
The first latency decision happens before retrieval or inference. The system has to determine whether a pause means "I am done" or "I am still thinking."
A fixed silence timer is easy to implement, but making it consistently natural is difficult. If the system waits too long, every response begins with unnecessary dead air. If it commits too quickly, it can cut users off before they finish dates, product names, email addresses, corrections, or multi-clause questions. Reducing endpointing latency without measuring false endpoints therefore risks replacing one poor experience with another.
Endpointing should be evaluated using paired performance metrics, including end-of-turn decision delay, false endpoint rate, transcript correction after the endpoint, and results segmented by intent, language, device, and network conditions. A short confirmation may support aggressive endpointing, while a technical question or email address may contain pauses that should remain part of the same turn.
Partial transcripts can also be useful for speculative routing or retrieval, provided that the work is inexpensive to cancel. The final transcript should remain authoritative whenever the user's last words change the meaning of the request. This allows the system to use streaming information to save time without sacrificing correctness.
The objective is therefore not simply to detect silence as quickly as possible. It is to determine, with sufficient confidence, when the user has actually yielded the floor.
2. Retrieval: Protect the Middle of the Turn
Retrieval is one of the places where an otherwise fast Voice AI system can lose a significant portion of its latency budget.
A hosted retrieval request is more than a search operation. Depending on the architecture, it can involve connection acquisition, authentication, load balancing, queueing, network transit, query execution, serialization, and deserialization. The Production AI Stack describes this as a retrieval latency tax: infrastructure overhead incurred before the model receives the context it needs. If an agent makes multiple dependent retrieval requests, that overhead can be paid multiple times within a single turn.
In a chat application, several hundred milliseconds of retrieval latency may be difficult for a user to notice. In Voice AI, the same delay becomes part of the silence between turns. When grounding consistently takes hundreds of milliseconds, teams may respond by skipping retrieval for requests that appear simple or by putting more information directly into the prompt. The first approach increases the risk of ungrounded answers, while the second can make prompts larger and less selective.
An alternative is to keep an already-loaded index in the agent runtime so that retrieval does not require a network request on every turn.
In the benchmarks published with the Production AI Stack, moving retrieval in process reduced median latency from 67 milliseconds to 5 milliseconds and P99 latency from 222 milliseconds to 13.5 milliseconds. In a published 100,000-document benchmark at top-k five, including embedding inference, Moss measured 3.1 milliseconds at P50 and 5.4 milliseconds at P99. These are published benchmark figures rather than guarantees for every corpus, machine, filter, or concurrency level, so teams should measure their own workloads.
The broader lesson is that retrieval performance is not just about making search faster. Lower and more predictable retrieval latency creates additional headroom for every other part of the Voice AI pipeline.
3. LLM Inference and TTS: Optimize the First Speakable Phrase
Time to first token is an important LLM metric, but it is not the same as the time at which a voice agent can actually begin responding.
The first token may be punctuation, an incomplete fragment, or language that is not yet stable enough to synthesize naturally. For Voice AI, there are therefore two connected budgets: the time required for the model to produce useful, stable language and the time required for speech synthesis to turn that language into playable audio.
Streaming between these stages is critical. Rather than waiting for the entire model response, the system can pass sufficiently stable output into speech synthesis and begin producing audio while the rest of the response is still being generated. However, chunk size needs to be tuned carefully. Very small chunks can reduce apparent latency while producing unnatural or fragmented speech, while large chunks introduce a buffer that effectively recreates the latency the system was trying to eliminate.
For this reason, teams should measure time to first token, time to first stable speakable phrase, and time to first audio separately. A model dashboard can report excellent time-to-first-token performance while the user is still waiting because the application has not yet produced enough stable language to speak.
4. Transport and Playback: Stop the Clock at the Speaker
A speech synthesis provider returning audio is not the same thing as the user hearing that audio. The response still needs to travel across the network, reach the client, enter a playback buffer, be decoded, and begin playing without immediately stalling.
This final stage is easy to overlook because most infrastructure dashboards stop measuring before the user experience actually begins. Voice AI systems should instead measure client receipt, buffer time, decode time, and first playback, along with the region and network conditions affecting those measurements.
Keeping unavoidable services in compatible regions and reusing connections can help reduce transport overhead, but the most important architectural decision is simply to measure the complete path. Provider-level timings explain individual components; microphone-to-speaker timing explains the experience.
Keep the Voice AI Turn Path Small
Many of the biggest latency improvements come not from making every component marginally faster, but from removing unnecessary work from the repeated turn path altogether.
Static assets should be built and validated before serving traffic. Reusable clients, connections, and models can be prewarmed at worker startup. Tenant configuration, knowledge, and conversation state can be loaded at session start rather than repeatedly on every turn. The critical path can then remain focused on the work that actually needs to happen for each interaction: endpointing, turn-specific retrieval, generation, synthesis, and playback.
This distinction is particularly important for first-turn latency. Cold-start work hidden inside the first request is still user-facing latency, even if the system looks fast once it is warm. First-turn and warm-turn distributions should therefore be reported separately rather than allowing a healthy steady-state median to conceal a poor first impression.
Streaming also needs to extend across every boundary. A single batch-oriented stage can erase the gains produced by streaming everywhere else in the stack.
There is a second critical path to consider as well: interruption handling. While the agent is speaking, the system must detect meaningful user speech, stop playback, cancel generation and synthesis, and preserve only the words the user actually heard. Interruption stop time should therefore be measured alongside false interruption and false barge-in rates. A system that responds quickly but treats every cough or background noise as a new turn is not necessarily delivering a better conversational experience.
Founding Agent as a Practical Example
Moss's Founding Agent, a voice landing-page assistant, applies these same latency principles in a real product experience.
Streaming speech recognition handles the turn decision, while deterministic navigation and scroll intents can bypass a model call altogether. Substantive questions retrieve context from an in-process Moss index, and stable language streams into speech synthesis. Tenant configuration and the knowledge index are loaded at session lifetime rather than inside every individual turn.
Each of these choices protects a different part of the latency budget. Some allow work to begin at the right moment, some eliminate unnecessary work, some remove network hops, and others keep audio moving through the pipeline. Together, they demonstrate why Voice AI latency is fundamentally an architectural problem rather than a single-provider optimization problem.
Measure the Conversation, Not the Provider Dashboard
A useful Voice AI trace should follow each turn from microphone to speaker using a shared identifier. At minimum, the trace should include endpoint decision time, final transcript time, route selection, retrieval wall time, prompt assembly, model time to first token, time to first stable speakable phrase, speech synthesis first byte, client receipt, buffer time, first playback, and cancellation timestamps for interrupted turns.
Each stage should have an owner, a measurement boundary, and a tail-latency objective. P50, P95, and P99 should be aggregated by region, device class, network type, and turn number. First-turn latency should be separated from warm-turn latency, model time to first token from first speakable phrase, and synthesis first byte from first audio actually played.
Those measurements also need to be considered alongside conversational quality. False endpoint rate, false barge-in rate, transcript correction rate, first-turn cold-start rate, and the percentage of substantive answers that were grounded all affect whether the user perceives the system as working well.
A latency improvement that produces more interruptions or fewer grounded answers is not necessarily an improvement. It may simply have moved the problem somewhere else in the stack.
The Best Latency Is Invisible
There is no single provider that can make a Voice AI system feel conversational on its own. A faster language model cannot recover a slow endpoint decision. Fast speech synthesis cannot speak context that is still crossing a retrieval boundary. And a low median latency cannot compensate for repeated tail-latency stalls.
The entire stack has to work within one small budget.
That means defining time to first audio as an end-to-end metric, assigning every stage a share of the budget, streaming across boundaries, moving repeated setup out of the turn, and removing unnecessary network work from the critical path. It also means measuring latency alongside the correctness of turn-taking, because speed without conversational accuracy does not produce a better voice experience.
When those pieces work together, the user asks a question, yields the floor, and hears a grounded answer when they expect one. The system does not need to explain that it is still working because the response arrives naturally enough that the user never has to wonder.
There is no second "Hello?" because there is no gap to explain.
The best Voice AI latency is the latency the user never notices.