Making voice agents sound human with expressive mode

Voice agents have gotten very good at knowing what to say. And with Gemma 4 31B on LiveKit Inference, they can say it with sub-second latency, fast enough that responses feel immediate.

But what about how they say it? Most voice agents today deliver every sentence in the same register: energetic and upbeat, but detached from what the user is actually feeling. If you tell an agent your flight was cancelled and you're stranded overnight with two kids, it shouldn't confirm your rebooking in the same enthusiastic tone it uses to upsell you on seat selection.

This isn't just an aesthetic problem. AI is great on analytical, "thinking" tasks but falls short on emotionally charged, "feeling" ones. A 2025 study in the Journal of Business Research called it the "feeling skills gap," and it has a cost: when a voice agent handles a customer's problem, customers come away less satisfied and less likely to buy again. Those are exactly the moments where matching the caller's emotional state matters most.

We've been working on this problem for a while. Today we're excited to announce expressive mode in LiveKit Agents. It's a single flag that makes your agent sound emotionally alive instead of flat.

Try the live demo#

Talk to it like a friend. Tell it some good news, or some bad news, and listen to how the delivery changes. Toggle expressive mode off to hear the same agent flat, or switch the voice to compare providers.

How expressive mode works#

Modern TTS models from providers like Fish Audio, Inworld, Cartesia, and xAI can render very lifelike prosody with warmth, excitement, hesitation, laughs, and sighs, but only if they're told to. That direction comes in the form of inline markup tags, and each provider speaks its own dialect.

Expressive mode acts as a translation layer between your LLM and your TTS. When enabled, the framework injects provider-specific instructions into the LLM prompt, teaching it to emit expressive markup inline with its response. This enables emotions, delivery styles, non-verbal sounds, and pauses, all chosen from the conversation context.

The LLM writes one marker dialect, and the framework lowers it to whatever the active provider understands:

1
<expr type="expression" label="excited"/> Oh my god, no way! <expr type="sound" label="laughing"/>
2
<expr type="expression" label="happy"/> I <expr type="prosody" label="emphasis">knew</expr> you'd nail it.

You never write these tags yourself. The LLM generates them, and the framework normalizes common mistakes (like a tag that isn't self-closed) before they reach the TTS. The tags are also stripped from the user-facing transcript, so your chat UI only ever sees clean text.

The second piece is less obvious. Streaming text to the TTS one sentence at a time turns out to work against expressiveness, because the TTS loses the emotional through-line of the response. Each sentence arrives without context, and the rendered emotion and pitch drift from one sentence to the next. Instead, we batch sentences into larger chunks before synthesis. We found this promoted contextual awareness in the TTS and was better at stabilizing emotion and pitch for each turn. These larger chunks don't add meaningful latency, especially if you're using a voice-optimized LLM like Gemma 4 31B.

Using expressive mode#

Enabling it is one parameter on your AgentSession:

1
from livekit.agents import AgentSession, inference
2

3
session = AgentSession(
4
    stt=inference.STT("deepgram/nova-3"),
5
    llm=inference.LLM("google/gemma-4-31b-it"),
6
    tts=inference.TTS("fishaudio/s2.1-pro", voice="9a9cf47702da476aa4629e2506d4a857"),
7
    expressive=True,
8
)

That's the whole setup. The agent picks its own delivery from the conversation, so most agents need nothing else.

Expressive mode works with any LLM. On the TTS side, it requires a LiveKit Inference model that declares a markup dialect. Fish Audio (fishaudio/s2.1-pro), Inworld (inworld/inworld-tts-2), Cartesia (cartesia/sonic-3), and xAI (xai/tts-1) are tuned for it today, with more providers on the way. A provider without a dialect synthesizes normally and the flag stays inert.

Steering delivery#

expressive=True gives you the full expressive range. Most production agents want that: the injected instructions already tell the model to match its delivery to the register of the moment, and to stay composed when the moment is heavy.

When you do need to take something away, pass an ExpressiveOptions dictionary instead of True:

1
from livekit.agents import AgentSession, ExpressiveOptions
2

3
expressive: ExpressiveOptions = {
4
    "speech_steering": {
5
        "pace": "slow",
6
        "nonverbal_sounds": {"laughing": False},
7
    },
8
}
9

10
session = AgentSession(
11
    # ... stt, llm, tts
12
    expressive=expressive,
13
)

Every key is a sparse override, so the example above keeps the full sound vocabulary except laughter. You can also replace or extend the injected prompt entirely with tts_instructions_template or tts_instructions_append.

Driving your UI from the agent's emotional state#

If your agent has a visual presence like a visualizer or a mood indicator, you'll want the frontend to know how the agent is feeling, not just what it's saying.

When the framework strips expressive markup from a transcript segment, it attaches the segment's leading delivery tag as an attribute named lk.expression, published on the lk.transcription text stream alongside the clean text. The value is a JSON object like {"value": "speak happy"}, so we can add fields later without breaking your parser.

One thing to know: that value is not a fixed vocabulary. Fish Audio emits single words from a closed set, Inworld emits free-form English like "soft, with genuine care", and models drift outside whichever set they were given. So rather than switching on the raw string, Agents UI ships a useExpression hook that matches it down to a normalized mood and hands you a color:

1
import { useAgentExpression } from '@livekit/compontents-react';
2
import { AgentAudioVisualizerAura } from '@/compontents/agents-ui';
3

4
export const MOOD_COLORS = {
5
  angry: '#F5222D',
6
  excited: '#FF7A45',
7
  happy: '#FFC53D',
8
  playful: '#F759AB',
9
  surprised: '#B37FEB',
10
  anxious: '#D46B08',
11
  hopeful: '#52C41A',
12
  empathetic: '#36CFC9',
13
  curious: '#6600FF',
14
  sad: '#2F54EB',
15
  calm: '#1FD5F9',
16
};
17

18
function Expressive() {
19
  const { mood } = useAgentExpression();
20
  const color = MOOD_COLORS[mood] ?? MOOD_COLORS.calm;
21

22
  return (
23
    <>
24
      <AgentAudioVisualizerAura color={color} />
25
      <span style={{ color }}>{mood}</span>
26
    </>
27
  );
28
}

label is always the provider's raw string if you want it, and the color is blended over a primary you choose, so the visualizer still reads as your brand. That's exactly what's driving the demo above.

Looking ahead#

This is our first release of expressive mode, and we're already working on what's next: mid-session control for switching delivery on the fly, tuned support for more TTS providers, and richer expression signals for frontends.

Try it today#

The fastest way to hear the difference is the demo above. Then enable it on your own agent with one line. The full demo agent is on GitHub and the expressive mode docs cover the rest.

Voice AI that gets the words right but the feeling wrong still loses users. Expressive mode is our first step toward agents that respond to how you're doing, not just what you said.

Give it a try! We'd love to hear what you build and any feedback you have.