The latency budget for a talking AI NPC, step by step
A talking AI NPC has, as a rough estimate, one to three seconds between the moment the player stops speaking and the first word the character says. Even with streaming at every stage, the range is about one to three seconds. Most of the time goes to deciding that the player has finished, waiting for the model’s first tokens, and producing the first chunk of synthesized voice.
This article walks through one spoken turn stage by stage. Every time range marked as an estimate is the writer’s, not a vendor figure. Replace each one with measurements from your own build. Rick Vu, who directed this piece, builds Red Dog Studio, a game studio building AI-native NPCs. Nothing here describes Red Dog’s pipeline or its numbers. The dollar side of the same pipeline is in what an AI NPC costs per player, and this article does not repeat that cost model.
Why the bar is high
People in conversation leave very little silence between turns. A 2009 PNAS study of turn-taking in 10 languages by Tanya Stivers and colleagues found “a minimization of silence between conversational turns” in every language it examined. Average gaps differed by language, but only within 250 milliseconds of the cross-language mean. A pipeline with a network round trip and three models in a row will not match a human reply. The real design question is what the player sees and hears while the pipeline runs.
These targets are opinions, not measured thresholds. Something on screen should react within about 0.3 seconds. The real spoken line should start within about two seconds. Typed dialogue can tolerate more, because players already read text at their own pace.
The budget for one spoken turn
The table follows one turn through a cloud pipeline that streams at each stage. The clock starts when the player stops talking and stops at the first audible word. Every time range in the table is an estimate.
| Stage | Estimated time | What sets it |
|---|---|---|
| End-of-turn detection | 0.3 to 0.7 s | How long the system waits in silence |
| Final speech-to-text | 0.1 to 0.3 s | The streaming model and the network |
| Prompt assembly | 0.01 to 0.1 s | Memory lookup and game state |
| Network and first token | 0.3 to 1.0 s | Region, provider queue, prompt, model size |
| First sentence of output | 0.1 to 0.3 s | Generation speed and sentence length |
| Text-to-speech first audio | 0.1 to 0.4 s | The voice model and the network |
| Lip sync start | under 0.1 s | The animation buffer |
| Total | about 0.9 to 2.9 s |
Two rows deserve attention. The first stage can cost as much as the language model, and it happens before any model runs. The full reply is missing from the table on purpose, because with streaming the voice starts long before the model finishes.
Detecting that the player has finished
The pipeline cannot answer until it decides the player has stopped. Voice activity detection makes that call, usually by waiting for a stretch of silence. OpenAI’s voice activity detection guide shows an example configuration with silence_duration_ms set to 500 and says that with shorter values, turns are detected more quickly. A shorter wait also cuts in on players who pause mid-thought. The same guide describes a semantic mode that judges from the words whether the speaker is done, with an eagerness setting that controls how long it will wait.
Games have an option that phone assistants lack. Push-to-talk ends the turn the instant the player releases the button, so the silence wait drops to zero. It costs some immersion, and it removes an estimated 0.3 to 0.7 seconds with no change to any model. Typed input skips both this stage and speech recognition.
Speech recognition and prompt assembly
Streaming speech recognition transcribes while the player talks, so little work remains when they stop. ElevenLabs’ models page lists Scribe v2 Realtime at about 150 milliseconds for partial transcripts, excluding application and network latency. NVIDIA’s post on the In-Game Inferencing SDK lists Whisper speech recognition among its local models, which removes the network trip for this stage.
Prompt assembly is ordinary game code. Reading game state takes almost no time. Memory retrieval is slower if it calls a remote vector database on every turn. Start retrieval from the partial transcript instead of the final one, and keep the memories of nearby NPCs in a local cache. How AI NPCs work covers what goes into that prompt.
Time to first token
Time to first token covers the network trip, any queue at the provider, and the model reading the prompt. OpenAI’s latency optimization guide says that cutting half of a prompt may improve latency by only 1 to 5%, so a shorter prompt is a weak lever on its own. Prompt caching helps the same stage. OpenAI’s prompt caching guide says caching can “Reduce the time spent processing input before the response starts,” though it gives no percentage. For the cache to apply, keep the persona and rules at the start of the prompt and put the player’s new line last.
Model size matters more. The latency guide says smaller models “usually run faster (and cheaper)”. Reasoning settings matter too. A model that thinks before it speaks adds every thinking token to the wait, so turn reasoning off or to its lowest level for live dialogue.
Every extra request in the chain pays this cost again. “Make fewer requests” is one of the guide’s seven principles, and “Parallelize” is another. Ask for the spoken line and the requested game action in one call. Run the check on the player’s input at the same time as generation, and throw the reply away if the check fails.
Output only has to stay ahead of the voice
The latency guide calls generating tokens “almost always the highest latency step” and says that cutting 50% of output tokens may cut about 50% of latency. That holds when you wait for the whole reply. With streaming, the voice can start on the first complete sentence. After that, the rest of the reply only has to arrive faster than the character speaks it.
That is a low bar. As an estimate, a character speaks about 150 words a minute, which is roughly three tokens per second. As an estimate, any model fast enough for live dialogue generates many times faster than that. Once the first sentence is out, generation speed stops mattering for a spoken NPC.
So the lever is the length of the first sentence. “The bridge is out.” starts sooner than a reply that opens with a forty-word clause. Ask for a short opening sentence in the persona prompt, and cap total output length, which also caps cost. Send text to the voice model one sentence at a time.
Guardrails need the same treatment. An output check that waits for the whole reply gives back everything streaming saved. Check each sentence before it goes to the voice, and keep an authored fallback line ready for when a check fails.
Voice and face
Text-to-speech adds its own time to first audio. ElevenLabs’ models page lists Flash v2.5 at about 75 milliseconds and Eleven v3 Conversational at about 280 milliseconds, both excluding application and network latency. The voice model you choose can move the budget by a fifth of a second before the network adds anything. Listen to both kinds of voice model on your own lines before you choose.
Lip sync follows the audio. NVIDIA’s In-Game Inferencing SDK post says “Real-time AI-driven animation such as Audio2Face delivers accurate lip sync”. As an estimate, driving the face from the audio stream adds under 0.1 seconds, as long as animation starts on the first audio chunk instead of the finished clip.
On-device inference trades the network for the GPU
Running models on the player’s machine removes the network round trip and the provider’s queue from every stage. The In-Game Inferencing SDK post says the SDK supports both on-device inference, on the CPU or the GPU, and cloud-based AI workflows. It lists Mistral-Nemo-Minitron models at 2B, 4B, and 8B parameters. NVIDIA’s announcement of ACE autonomous game characters says the 2B model fits in as little as 1.5GB of VRAM.
The cost is GPU time. The SDK post says “effective GPU scheduling is crucial to maintain frame rates” when AI runs alongside rendering. A consumer GPU that is also drawing the frame may generate tokens more slowly than a data center. As an estimate, on-device inference usually wins on time to first token, where the network hurts most, and can lose on raw generation speed. Spoken output only has to outrun the voice, so that trade often favors the device for short lines. A split is a sensible default. Run speech recognition and short reactions locally, and send long or rare scenes to a larger cloud model.
Hide what you cannot cut
“Make your users wait less” is another of the guide’s principles. In a game, the simplest version is an authored reaction that plays the moment the player stops talking. It can be a nod, a glance, or a recorded “Hm” in the character’s own voice, picked by a rule in game code. It needs no model call, so it fits inside the 0.3 second target. Write several reactions per character so players do not hear the same one twice in a row.
Some turns need no model at all. The guide’s last principle is “Don’t default to an LLM.” Greetings, shop menus, and combat barks can come from authored lines. Save the model for turns where an open answer changes the scene.
Measure each stage in your build
The ranges in this article are estimates, and your build will differ. Log a timestamp at every stage boundary on every turn. Track the median and the 95th percentile per stage, because a player remembers the one slow turn more than the average. Then spend effort on the largest row first, which is often the silence wait and not the model.
Related reading
- What an AI NPC costs per player, for the dollar budget behind the same model choices.
- How AI NPCs work, for what the prompt holds and where the checks sit.
- Keeping AI NPCs in character, for the guardrails that now run on each streamed sentence.
Questions
How fast does an AI NPC need to reply?
People leave very little silence between turns, so no cloud pipeline matches a human reply. A reasonable target, which is an opinion and not a measured threshold, is a visible or audible reaction within about 0.3 seconds and the real spoken line within about two seconds. The quick reaction can be an authored nod or sound that needs no model.
What adds the most latency to an AI NPC?
By estimate, two stages cost the most. The wait to decide that the player has finished speaking and the model's time to first token can each take 0.3 seconds or more. Generating the full reply is the slowest step if you wait for it, but streaming hides it because the voice can start on the first sentence.
Does on-device inference make NPCs faster?
It removes the network round trip and the provider's queue, which mostly shortens the time to first token. The model shares the GPU with rendering, so it may generate tokens more slowly than a data center and can cost frame rate. For short spoken lines, that trade, as an estimate, often favors the device.
How do games hide model latency?
The game plays an authored in-character reaction, such as a nod or a recorded sound, the moment the player stops talking, and then streams the real line. Push-to-talk removes the silence wait. Simple turns like greetings and shop menus can skip the model entirely.