Debugging short utterances ("yes", "ok") in voice agents

It is challenging to build a voice agent that responds reliably and naturally to short answers: "yes", "no", "ok", "42". How do you know when the user has finished speaking? Is the user answering you, or just agreeing with you?

This article covers the most common causes of short-utterance failures in production, and how to fix them. It doesn't apply if you use manual turn control, where your application decides when each turn ends.

User symptoms#

To the user, it feels like the agent is ignoring the answer:

1
Agent: Do you like toast?
2
User:  No
3
Agent: <silence>

Often the user tires of waiting and reprompts the agent, and the conversation gets back on track:

1
Agent: Do you like toast?
2
User:  No
3
Agent: <silence>
4
User:  No!
5
Agent: How about a muffin?

Either way, the experience is poor and unnatural.

What causes agent silence after a short utterance?#

A voice agent stays quiet until it decides the user has finished speaking. That decision is called turn detection, and when it fails for short utterances there is rarely a single root cause. Much depends on how your agent is built.

It helps to understand how the agent detects the end of a user's turn, or end of utterance (EOU). The diagram below is simplified, but shows what makes up the user_turn span:

For clarity, the diagram omits:

• The tunable options and VAD settings that can control event timings.
• Exactly how the endpointing delay is calculated from your endpointing and silence settings. Note that it is measured from the end of speech, not from the moment EOU detection finishes.
• When EOU prediction begins, triggered by the VAD after a sufficient period of silence.
• The fact that you may receive multiple (or zero) interim transcripts, and that with turn_detection="stt" the end of speech indicator comes from the STT rather than VAD.
• The fact that turn_detection="vad" does not run EOU detection at all.

The important takeaway from the diagram is that EOU requires both the final_transcript and the end_of_speech. Interim transcripts do not close the turn. If you see a very long user_turn span compared to how long the caller actually spoke, it is a good indication that one or both of these events never happened. In your agent, another indication is when on_user_turn_completed never fires for that turn.

The rest of this article works through the common root causes, and what to do about each one.

STT does not generate a final transcript#

Many STT providers fail to detect short sections of speech, and when that happens no final_transcript is sent. Sometimes an interim_transcript arrives and the final_transcript still does not, which fails in exactly the same way: without a final_transcript the turn never closes, and the user gets the silence shown above.

This is not specific to any one provider and we have seen the same issue across multiple providers and models. Since final_transcript is a prerequisite for turn detection, switching your turn detection model is unlikely to improve this issue.

Many customers experiencing this issue experiment with different providers and models. There is no single best STT provider for short utterances, because the answer varies with your use case, language, user environment, and the continual improvements in new models. A good place to start is the models recommended out of the box with our agent starter, as well as the other STT models offered through LiveKit Inference.

Graceful recovery#

The transcription_timeout setting, new at the time of writing, allows graceful recovery when the final_transcript does not arrive within a specified number of seconds. It is opt-in. Once enabled, you can prompt the agent to ask the user to repeat themselves.

1
from livekit.agents import AgentSession, UserTranscriptionTimeoutEvent
2

3
session = AgentSession(
4
    ...
5
    # VAD heard speech but no transcript landed within 5s of the user stopping.
6
    transcription_timeout=5.0,
7
)
8

9
@session.on("user_transcription_timeout")
10
def _on_transcription_timeout(ev: UserTranscriptionTimeoutEvent) -> None:
11
    # Optionally, require the speech to be over a certain length.
12
    if ev.speech_duration < 0.4:
13
        return
14

15
    session.generate_reply(
16
        instructions="Tell the user you didn't catch that and ask them to repeat it.",
17
    )

Alternatively, some developers build their own graceful recovery mechanism around user_away_timeout.

Overly aggressive endpointing#

The endpointing delay is how long the session waits after the user stops speaking before it commits the turn. It is measured from the end of speech, and it is either min_delay if the turn detector thinks the turn is complete, or max_delay if it thinks the turn is incomplete. It is tempting to reduce these significantly (especially max_delay) in the hope of catching short utterances. Avoid this. A short endpointing delay forces the turn to end prematurely, and in extreme cases the final_transcript is attached to the wrong turn.

If your agent responds prematurely, answers only the first half of a question, or produces transcripts that are out of order, increase your endpointing delay. If you have overridden the framework defaults, try removing your overrides first.

One caveat: if you are using turn_detection="stt", set endpointing.min_delay to 0 to avoid stacking delays.

Overly permissive endpointing#

Your STT may capture the transcript correctly, but the turn does not close quickly, leaving the user waiting for the agent to respond. The delay is most obvious after a short reply.

Two situations make this worse:

• VAD-only turn detection has no way to take account of what was said, so it always waits out the longer of the VAD's min_silence_duration and the session's endpointing.min_delay. If either value is high, every turn is noticeably longer.
• A turn detector that fails to recognize a complete turn after a short utterance falls back to waiting out endpointing.max_delay, so a large value there becomes a long silence.

Noise cancellation silences user audio#

LiveKit provides noise cancellation that can run at several points in your solution: in your client, in your agent, or on your SIP trunk. Configuring it correctly for your use case matters, and this post on "Noise cancellation: what it is and how it works" is a comprehensive overview.

Misconfigured noise cancellation drops audio, because the user's short speech is classified as noise. Even correctly configured noise cancellation can drop short utterances, especially for SIP clients, depending on network quality and how loud the user is.

If you suspect noise cancellation is suppressing your user's audio, follow these steps:

1. Be sure you are not stacking your noise cancellation models, for example applying an "enhanced" model on both your SIP trunk and your agent. For more information, refer to the noise cancellation post.
2. Be sure you are using the correct noise cancellation model: voice isolation for 1:1 conversations, and background noise suppression for group conversations. If you find voice isolation too aggressive, there is no harm in trying background noise suppression for 1:1 conversations instead.
3. As a test, disable noise cancellation in your agent and observe whether the situation improves.
4. If your agent performs well with noise cancellation disabled, consider leaving it disabled. Some STT models are trained on noisy audio and cope well in similarly noisy production environments.

Poor telephony quality#

If short phrases are more likely to be lost over the telephone than in your other tests, check whether your SIP connection is a contributing factor.

Where possible, use a higher-quality codec, ideally one that supports HD voice such as G.722, which operates at 16kHz. Not all SIP providers support HD voice, though it is available through LiveKit Phone Numbers.

Also, be sure you are routing your SIP connection sensibly. Your SIP users enter the LiveKit network at the point closest to your SIP provider's trunk, so check that the trunk is configured in the right region. Getting this right reduces round trip time and SIP jitter, and improves your overall agent latency.

Misclassified backchannels#

Backchanneling is when the user responds with cues such as "uh-huh", "okay", or "right" that show they are paying attention but do not need a response. Backchannels are usually short utterances, and the agent can mistake them for an interruption. That will not make the agent go silent, but it does disrupt the conversation flow.

To stop backchannels from being labeled as interruptions, you have two options:

• Use LiveKit adaptive interruption handling (recommended), which automatically recognizes backchannels and maintains the conversation flow.
• Raise the interruption.min_duration setting, though doing so also makes your agent less likely to detect genuine interruptions.

Note that the related setting, interruption.min_words, has a potential side effect. If the user replies while the agent is still speaking and the reply is shorter than min_words, the turn is not committed at all, which produces exactly the silence this article is about.

Diagnosis checklist#

Work through these steps in order:

1. Be sure you are on the latest version of LiveKit Agents and its dependencies. We make continual improvements and fix issues in the framework as they arise.
2. Enable Agent Observability so that you have a record of what went wrong in failing sessions.
3. Look for common patterns in failing sessions. If you consistently see unexpectedly long user turns for short utterances, add graceful recovery.
4. Check that your noise cancellation settings are correct, since misconfiguration there affects more than just short utterances.
5. If you see more issues with SIP clients than with other clients, investigate your telephony quality.
6. Use the correct turn detector model for your use case, following the advice in the Configuring Turn Detection guide, then set your endpointing and related delays with the turn-taking tuning guide.
7. If you still see issues, evaluate other STT models and providers to find one that works better for your application.