The 300ms Threshold

The silence after you speak to an AI. This book treats 0.3 seconds as a rough point where that gap starts to feel wrong.

“Hello.”

Yu tried to sound as natural as he could. It was the first demo of their voice AI product. After six months of development, the day had finally come to present it to investors.

Silence.

One second. Two seconds.

“Did it freeze?”

Yu will never forget the moment the investor’s expression changed.

The AI did answer. “Hello, is there anything I can help you with?” The synthesized voice sounded natural. The reply itself was perfect. But it came too late.

That night, Yu stayed behind in the office with his MacBook open in front of him. A Google Sheets spreadsheet listed the measurements.

1.8 seconds. 2.1 seconds. 1.9 seconds.

Every test case was over 1.5 seconds.

“This isn’t a conversation.”

He said it to no one, and the words hung in the quiet office.

That was the moment Yu’s pursuit of 300ms began.

This book has two parts woven together: the story of Yu, a voice AI engineer, and Misaki, a UX designer, as they fight latency in pursuit of human-like conversation, and explanations of the technology behind that fight. Yu and Misaki are fictional. The values they measure, such as 1.8 seconds or 350ms, are measurements they made within the story. I keep those apart from the numbers I present as general findings, and every general number comes with the name of its source. Full citations and URLs are in the references at the end of each chapter and at the back of the book.

Throughout the book I draw on what I learned in the field: many years building real-time communication products on WebRTC, and later designing and building conversational AI products. The technical details and vendor comparisons for OpenAI Realtime API, Gemini Live API, Pipecat, LiveKit, and others are based on information available when I wrote the book in March 2026, and I rechecked the sources for the September 2026 revision. Specs and numbers in this field change quickly, so check the latest official documentation before you adopt anything.

In conversation between people, the gap between one speaker finishing and the next one starting clusters around 200 milliseconds (Chapter 1). Meanwhile, a 2025 survey of turn-taking research (Castillo-López et al.) cites earlier work to say that current spoken dialogue agents wait 700 to 1,000 milliseconds before they start speaking. The reason is that many systems wait for silence to last a set amount of time (700 milliseconds, for example) before they decide the user has finished. That is three to five times the human gap. This book treats that difference as the center of the “something feels off” problem in voice AI.

Following Yu and Misaki, and mixing in my own work experience and recent technical developments, the book systematically examines latency in voice AI from these angles:

• How fast does human conversation move? (Chapter 1)
• At what delay does the experience fall apart? (Chapters 2-4)
• Where does the delay come from? (Chapters 5-6)
• How do you make it faster? (Chapter 7)
• When you can’t make it faster, how do you cover for it? (Chapter 8)
• How do you avoid both interrupting and lagging? (Chapter 9)
• What can we learn from existing voice assistants? (Chapter 10)
• How do you get past the 300ms wall with edge AI? (Chapter 11)
• How do you turn all of this into a checklist to use during implementation? (Chapter 12)

The table below lists the main numbers in this book. They come from three kinds of sources: values observed in research, values published by vendors, and this book’s design targets. The “Type” column tells them apart.

This book’s primary metric is the time from the end of the user’s speech to the first audible sound (E2E / TTFA). The clock starts when the user’s voice actually stops, so the time the system spends deciding “the user has finished” is part of this interval. Measure this interval on your own system before you apply any of the design targets.

0.3 seconds is the number this book sets as its design goal. Together with Yu and Misaki, we look for ways to fit end-of-turn detection, speech recognition, the language model, speech synthesis, the network round trip, and the start of playback into that short window.

By the end, you should be able to measure your system’s latency stage by stage, decide where to cut first, and decide how to cover whatever you can’t cut. I have written down Yu and Misaki’s stumbles as well, so you don’t have to take the same long way around.

The sound of Misaki setting down her coffee cup carried through the hushed office.

In the hushed office, the only sound is the small clink of a cup being set down.

“Yu, I have an idea.”

Yu still hadn’t recovered from the failed demo the day before. Misaki couldn’t bear to see him like that, so she spoke up.

“Why don’t we record a conversation between people and analyze it? Let’s use numbers to see how it differs from the AI.”

“We can’t start without data. Let me record the two of us talking this afternoon, too.”

A Gap Shared Across Languages

Part of the rhythm of human conversation is shared across languages.

In 2009, a research team led by Tanya Stivers, centered at the Max Planck Institute for Psycholinguistics, collected recordings of natural conversation in 10 languages (English, Japanese, Danish, Dutch, Italian, Korean, Lao, Tzeltal, Yélî Dnye, and ǂĀkhoe Haiǁom) and compared the timing of speaker switches (turn-taking). To keep conditions comparable, they looked only at questions that can be answered yes or no, and at the gap before the answer.

The result was clear. In every language, the most common gap before the answer (the mode) fell between 0 and 200ms. The mean across all the data was about 208ms (Stivers et al. 2009).

200ms. That’s 0.2 seconds.

That night, when Yu analyzed their own conversation, he was startled by what he found and had to face the facts. “It really is 200ms. Our AI takes 2 seconds. Ten times slower.”

The languages do differ. The per-language means ranged from about 7ms for Japanese, the shortest, to about 469ms for Danish, the longest. Even so, every language showed the same tendency to avoid overlap and keep silences short. Stivers and colleagues argue that the differences between languages probably feel bigger than they are because people are sensitive to even small timing differences.

What 200ms Means

Getting words out, on the other hand, takes time. In picture-naming tasks, where people see a picture and say its name, the time to start speaking is estimated at about 600ms on average, and 740 to 800ms for two nouns (as summarized by Levinson and Torreira 2015). Those figures were not measured in natural conversation, but they are more than three times the gap between turns.

If you only started preparing after the other person stopped, you could never start talking within 200ms. You have to begin preparing your reply before they finish.

Levinson and colleagues argue that listeners start building their reply while the other person is still talking, as soon as they can see where the speaker is going. It is a model of predicting while listening and preparing in advance. In this book I call this parallel processing thinking while listening. It is also the inspiration for the streaming design in Chapter 7.

“So humans run in parallel,” Yu muttered. “Our AI runs sequentially. Listen, think, speak, one at a time. No wonder it’s slow.”

People prepare their reply while the other person is still talking and start speaking in about 200ms. Yu’s sequential AI only started working after the user finished, and took about 2 seconds.

Gaps Longer Than 200ms

Not every turn switch happens at 200ms. Even in the Stivers data, the per-language means spread over several hundred milliseconds.

In the context of voice AI, Aaron Ng, a machine learning engineer at Speechmatics, wrote in a blog post that the typical gap between turns in human conversation is about 600ms. His view is that a gap of that length can come across as thoughtful and polite, a sign that the listener is taking in what was said. The same post says that a voice agent that answers 200ms after the user stops may be technically impressive but doesn’t sound natural. Read this as one company engineer’s opinion, not a research finding. The post doesn’t say how the gap was measured or what kinds of conversation it covered, and the number isn’t directly comparable to the Stivers mode.

So how long can a gap get before the listener starts to worry? None of the sources this book consulted puts that boundary at a single number. It depends on what’s being said, the relationship between the speakers, culture, and how the system decides that the user has finished speaking. For voice AI, I lay out working targets in Chapter 4 as “the three cliffs.”

Implications for Voice AI

This research gives voice AI designers three things to keep in mind.

1. 200ms is the reference point for conversation between people

At least for answers to yes/no questions, the most common gap clusters between 0 and 200ms regardless of language. Since preparing to speak takes 600ms or more, that speed is only possible with prediction and early preparation. If a voice AI aims for “natural conversation,” its distance from this reference point is one useful yardstick.

2. You need a design that thinks while listening

To aim for speeds in the 200ms range, you need a pipeline that prepares the response while it is still processing the user’s speech. Sequential processing that starts only after the user has completely finished won’t make it in time. The earlier you start preparing, though, the more often the system jumps ahead while the user is still talking. How far to look ahead depends on the use case and on how accurate your end-of-turn detection is (Chapter 9).

3. You can fill the gap, with conditions

When you can’t answer within 200ms, one option is to fill the gap with a filler (“Let’s see,” “Um”). But if the user meant to keep talking, the filler becomes an interruption. Use fillers when you’re confident the user has finished, and check their effect with user evaluations for each use case (Chapters 8 and 9).

The next morning, Misaki put a stack of printouts on Yu’s desk. “I found an old article by Jakob Nielsen. This is wild.”

References

• Stivers, T. et al. “Universals and cultural variation in turn-taking in conversation.” PNAS, 2009.
• Levinson, S. C. and Torreira, F. “Timing in turn-taking and its implications for processing models of language.” Frontiers in Psychology, 2015.
• Speechmatics. “Your AI Assistant Keeps Cutting You Off. I’m Fixing That.” 2025.

9 a.m. in the meeting room. Misaki had the printed article spread out on the table.

“Yu, do you know Jakob Nielsen? He’s one of the big names in UX. This comes from a 1993 book, and it still holds up.”

Yu was still rattled by yesterday’s 200ms shock. Misaki put a table of numbers in front of him.

“There are three walls: 100ms, 1 second, and 10 seconds. We’re already past 1 second, so we’re in the zone where the experience starts to break down.”

Yu narrowed his eyes. “Does this apply to voice UI too?”

“That’s what I want to find out.”

Applying a GUI Classic to Voice

In 1993, Jakob Nielsen laid out three thresholds for UI response time in his book Usability Engineering. An excerpt from Chapter 5 of that book is published by Nielsen Norman Group as the article “Response Times: The 3 Important Limits” (with a note added in 2014). More than 30 years later, it is still widely cited as a foundation of UX design.

These numbers rest on how people perceive and think, not on how fast computers are. Nielsen notes that the basic advice on response times has stayed about the same since Miller in 1968 and Card and colleagues in 1991.

In Voice UI, the Thresholds “Shrink”

In a GUI, a 1-second delay is acceptable, because the user can watch a loading indicator on the screen.

Voice UI is a different situation.

Speech disappears as soon as it’s spoken.

In a text chat, even if you wait a second, the screen shows “typing…”, and you can reread earlier messages while you wait. A voice-only UI has nothing that stays on a screen. You can ask the system to repeat itself, but while you wait there’s no cue you can check with your eyes. Silence is just silence.

“Right,” Yu nodded. “No screen, so no cues. A world made only of sound is tough.”

So in this book, I translate the GUI thresholds for voice UI as follows and use them as design targets. No study has re-tested the GUI thresholds on voice. Read this as this book’s hypothesis.

The 4-second figure is backed by one experiment. At ACM CUI 2025, Maslych and colleagues reported an experiment in which people held free-form conversations with an LLM-powered agent inside a VR space. They set the response delay to three levels, 1.5, 4.0, and 6.5 seconds, combined each with and without fillers, and had 54 people rate the experience. The authors concluded that delays of more than 4 seconds lower the quality of the experience, and reported that natural fillers (bridging the gap with a word like “hmm” plus a thinking gesture) improved how the response time was perceived in the 4.0- and 6.5-second conditions. They didn’t measure 4 seconds as an exact boundary. The conclusion comes from comparing three conditions: 1.5, 4.0, and 6.5 seconds.

This was an experiment with an embodied agent, though, so it may not carry over directly to a voice-only UI. The fillers also combined words with gestures, and the effect of a voice-only filler wasn’t measured on its own. And since the shortest condition was 1.5 seconds, the study says nothing about the range under 1 second. In this book I use 4 seconds as “the line before which you should already have done something.”

Misaki wrote the numbers on the whiteboard. “Now I can see the target,” Yu said quietly. “Somewhere between 300ms and 500ms, we return some kind of response. That’s the lifeline for voice UI.”

Another Baseline: The Doherty Threshold

In 1982, Walter J. Doherty and Ahrvind J. Thadhani of IBM reported in an internal technical report that when a computer responds within 0.4 seconds, user productivity rises sharply. This is known as the “Doherty threshold.”

The Doherty threshold is widely described this way: with a response under 400ms, users experience the action and the reaction as one unbroken flow, and the sense of “waiting” is less likely to arise. How people perceive waiting is the subject of Chapter 3.

The number originally came from studying productivity in terminal work. It wasn’t measured on voice, from the end of the user’s speech to the first audible sound. In this book I borrow 400ms as a target for returning the first reaction within that interval. My hypothesis is that if some reaction (a filler counts) is audible within 400ms of the user finishing, it gives the user a cue that they were heard. Whether 400ms is enough, or too short, is something to check in user testing for your own service.

For each of the four main targets (200ms, 400ms, 500ms, 4 seconds), decide what to return. The numbers are this book’s design targets.

Summary: Timing Design for Voice UI

The numbers in the next table are design targets applied to this book’s primary metric, the time from the end of the user’s speech to the first audible sound. That interval differs from what the original studies measured (until a GUI finishes processing, until a terminal responds, and so on), so treat all of them as hypotheses to confirm with user testing.

The two serve different roles: 300-500ms is the target for the first reaction, and 1 second is the warning line for when the actual answer still hasn’t started. Timing design for voice AI means deciding “at which stage do we return what” with these thresholds in mind. A design that waits for the perfect answer and returns it all at once will almost certainly go past 1 second.

For example: by 400ms, acknowledge with a short backchannel that you got the request. If you’re going to pass 1 second, say something like “Let me look that up” to show you’re working on it. Then start speaking the answer from whatever part is ready. Keep returning “something” in time for each threshold. That is this book’s approach to timing design.

“But theory alone won’t cut it,” Misaki said, putting down her pen. “Let’s test how users actually feel.”

References

• Nielsen, J. “Response Times: The 3 Important Limits.” Nielsen Norman Group, 1993 (note added 2014).
• Doherty, W. J. and Thadhani, A. J. “The Economic Value of Rapid Response Time.” IBM Technical Report GE20-0752-0, 1982.
• Maslych, M. et al. “Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual Agents.” CUI ‘25 (ACM), 2025.