Skip to main content
close
Field Guides

Using the Testing Framework with Realtime Models

Learn how to properly set up and test your agents using the LiveKit testing framework with both standard LLM models and realtime models.

Last Updated:

Guide

The LiveKit testing framework allows you to test your agents with both standard LLM models and realtime models. This guide covers how to properly set up and use the testing framework with realtime models.

Setting Up Realtime Model Tests

To test an agent with a realtime model, use the following pattern:

1
async with (
2
    openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
3
    AgentSession(llm=rt_llm, userdata=userdata) as sess,
4
):
5
    await sess.start(YourAgent(userdata=userdata))
6
    result = await sess.run(user_input="...")

This pattern uses Python's async context manager to properly initialize and clean up both the realtime model and the agent session.

Important Notes

When testing with realtime models, keep these key points in mind:

  • Use text modalities — When using realtime models, make sure to specify modalities=["text"] as the testing helpers are designed to work with text input and output.
  • Context manager required — The realtime model must be used within an agent session context manager to ensure proper resource cleanup.
  • Suppressing warnings — If you're seeing deprecation or resource warnings in Python 3.12+, you can suppress them with:
1
import warnings
2


3
warnings.filterwarnings("ignore", category=DeprecationWarning)
4
warnings.filterwarnings("ignore", category=ResourceWarning)

Basic Test Example

Here's a complete example of testing an agent with a realtime model:

1
import pytest
2
from livekit.agents import AgentSession
3
from livekit.plugins import openai
4


5
from your_agent import YourAgent, UserData
6


7
@pytest.mark.asyncio
8
async def test_agent_greeting():
9
    userdata = UserData()
10


11
    async with (
12
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
13
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
14
    ):
15
        await sess.start(YourAgent(userdata=userdata))
16
        result = await sess.run(user_input="Hello!")
17


18
        await result.expect.next_event().is_message(role="assistant").judge(
19
            rt_llm, intent="Provides a friendly greeting"
20
        )

Validating Function Calls

You can test that your agent correctly calls function tools:

1
@pytest.mark.asyncio
2
async def test_function_call():
3
    userdata = UserData()
4


5
    async with (
6
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
7
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
8
    ):
9
        await sess.start(YourAgent(userdata=userdata))
10
        result = await sess.run(user_input="Book an appointment for tomorrow")
11


12
        # Verify the function is called with expected arguments
13
        await result.expect.next_event().is_function_call(
14
            name="book_appointment",
15
            arguments={"date": "tomorrow"}
16
        )
17


18
        # Verify function output is processed
19
        await result.expect.next_event().is_function_call_output()

Testing Conversation Context

To test that your agent maintains conversation context across multiple turns:

1
@pytest.mark.asyncio
2
async def test_conversation_context():
3
    userdata = UserData()
4


5
    async with (
6
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
7
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
8
    ):
9
        await sess.start(YourAgent(userdata=userdata))
10


11
        # First turn - establish context
12
        result1 = await sess.run(user_input="My name is Alice")
13


14
        # Second turn - verify context is maintained
15
        result2 = await sess.run(user_input="What's my name?")
16


17
        await result2.expect.next_event().is_message(role="assistant").judge(
18
            rt_llm, intent="Correctly recalls that the user's name is Alice"
19
        )

Mocking Function Tools

For isolated unit tests, you can mock function tools to control their behavior:

1
from livekit.agents.voice.run_result import mock_tools
2


3
@pytest.mark.asyncio
4
async def test_with_mocked_tools():
5
    userdata = UserData()
6


7
    async with (
8
        openai.realtime.RealtimeModel(modalities=["text"]) as rt_llm,
9
        AgentSession(llm=rt_llm, userdata=userdata) as sess,
10
    ):
11
        await sess.start(YourAgent(userdata=userdata))
12


13
        # Mock the function to return a specific value
14
        with mock_tools(YourAgent, {
15
            "check_availability": lambda date: "Available at 2pm and 4pm"
16
        }):
17
            result = await sess.run(user_input="Check availability for Monday")
18


19
            await result.expect.next_event().is_function_call(name="check_availability")
20
            await result.expect.next_event().is_function_call_output(
21
                output="Available at 2pm and 4pm"
22
            )

Example Test Implementations

For complete reference implementations demonstrating these patterns in real-world agents, check out:

Summary

PatternUse Case
RealtimeModel(modalities=["text"])Testing with OpenAI realtime models
async with context managersProper resource management
result.expect.next_event()Asserting on agent responses
mock_tools()Isolated unit testing of function calls

Additional Resources

For more examples and advanced testing patterns, refer to our voice agents examples repository.

Was this helpful?

Read related documentation

Agents overview

Get started with voice AI agents

Quickstart guide

Build your first agent

Agent models

STT, TTS, and LLM plugins

Find more Agents guides

Building multi-agent architectures with LiveKit agents

Learn best practices for building multi-agent architectures including session state management, chat context handling, TaskGroup patterns, and dynamic per-client routing.

Can you increase agent deployment limits?

Understand the hard cap on agent deployments and how to build a multi-tenant agent that scales without provisioning more slots.
View all Agents guides →