Your Model Isn’t Bad at Tool Calling. Your Serving Stack Is.

Here's a failure mode that comes up in the community: a team picks an open model with excellent tool-calling benchmarks: Gemma 4, Nemotron 3, take your pick, wires it into a voice agent, and the agent never calls a single tool. Worse, sometimes it reads the tool call out loud:

1
Agent: "call colon finish underscore conversation farewell message..."

The model isn't broken. The framework isn't broken. What's broken is an assumption: that tool-calling ability lives entirely inside the model. It doesn't. It lives in the pairing of a model and the serving stack that hosts it, and the same model can score 100% behind one endpoint and 0% behind another.

This post explains where tool calls actually come from, why GPT models work everywhere while open models are provider-dependent, and how to diagnose the problem in about a minute.

Where a tool call actually comes from#

When your agent framework sends a request with a tools array, three separate systems have to cooperate before your function runs:

1. The model decides to call a tool and emits that decision in its own native syntax. Every model family has its own: OpenAI models emit OpenAI's JSON format, gpt-oss emits the "harmony" format, Gemma 4 emits something like <|tool_call>call:get_weather{location:<|"|>Tokyo<|"|>}<tool_call|>, and Nemotron 3 uses XML-ish tool tags when unparsed. This syntax is baked in during training.
2. The serving stack the inference server behind the API, parses that native syntax out of the raw token stream and translates it into the structured tool_calls field of the OpenAI Chat Completions response.
3. The agent framework reads the structured tool_calls field and executes your function.

LiveKit Agents, like most frameworks, only reads step 3's structured output. It deliberately never scrapes tool calls out of text content, text-scraping every model family's syntax client-side would be fragile and would break streaming. So the entire chain hinges on step 2.

When the serving stack has no parser for the model's format, the tag falls through as ordinary assistant text. The framework sees a normal chat message, hands it to TTS, and your agent recites markup to a customer.

Why GPT models work everywhere#

OpenAI models emit OpenAI's format natively, there's nothing to translate, so they work behind any OpenAI-compatible endpoint by definition. That's why teams conclude "only GPT models do tool calling well." What they're actually observing is that OpenAI-format models are the common case whose native syntax needs no extra parser on an OpenAI-compatible endpoint. Native-API plugins (Anthropic, Gemini, Bedrock Converse) do their own translation, which is why those paths also work when wired correctly.

Open models are a different story. Whether they work depends on whether that specific provider configured that specific model's parser:

• gpt-oss 120B works on Groq, Cerebras, and Bedrock because the harmony format is widely supported; every major provider ships a parser for it.
• Gemma behind one provider's OpenAI-compat layer fails while the identical weights behind another provider succeed, because one of them parses Gemma's tag format and the other passes it through as text.
• The published benchmark numbers are real, they were just measured against a serving stack that does the parsing.

One subtle point that trips people up: a well-formed tag appearing in your transcript is proof the request side is fine. The model saw your tool schema, picked the right tool, and produced valid arguments. Nothing about your tool definitions, decorators, raw schemas, descriptions, can fix what happens next, because the failure is on the response path.

Reasoning models fail twice#

Hybrid reasoning models like Nemotron 3 add a second failure mode on top of the first. These models emit a thinking block before the answer, and the tool call comes after (or worse, inside) the reasoning. The serving stack now needs two parsers configured: one to separate reasoning from the response, and one to extract the tool call. Miss either and tool calling silently dies, even on a provider that handles non-reasoning models fine.

That's why teams report reasoning models that are "great conversationally but never call tools": conversation only needs the text path to work; tool calling needs the whole parsing chain.

The standard mitigations:

• Turn reasoning off for voice. Most hybrid reasoning models expose a toggle, a serving parameter such as enable_thinking: false, or a system-prompt switch like /no_think on older Nemotron cards. Check the model card. A voice agent rarely benefits from long hidden reasoning chains, and it pays for them in latency.
• Configure both parsers if you self-host. On vLLM that's --reasoning-parser and --tool-call-parser, and both are model-family-specific (for Nemotron 3, that often means a pairing like qwen3_coder for tools).

LiveKit Inference strips <think>…</think> blocks from streamed text before they reach speech synthesis, and you can do the same in a custom llm_node on other providers, but no framework can recover a tool call the server never structured.

The one-minute diagnosis#

You don't need your agent, or LiveKit, to find out which side of the line you're on. Call the endpoint directly with a tool attached and a prompt that should trigger it:

1
curl https://your-provider.example.com/v1/chat/completions \
2
  -H "Authorization: Bearer $API_KEY" \
3
  -H "Content-Type: application/json" \
4
  -d '{
5
    "model": "your-model",
6
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
7
    "tools": [{
8
      "type": "function",
9
      "function": {
10
        "name": "get_weather",
11
        "description": "Get the current weather for a location.",
12
        "parameters": {
13
          "type": "object",
14
          "properties": {"location": {"type": "string"}},
15
          "required": ["location"]
16
        }
17
      }
18
    }]
19
  }'

Then look at where the call landed:

1
// ✓ Serving stack is doing its job
2
"message": {
3
  "content": null,
4
  "tool_calls": [{
5
    "type": "function",
6
    "function": { "name": "get_weather", "arguments": "{\"location\": \"Tokyo\"}" }
7
  }]
8
}
9

10
// ✗ No parser: the model's native syntax leaked into content
11
"message": {
12
  "content": "<|tool_call>call:get_weather{location:<|\"|>Tokyo<|\"|>}<tool_call|>",
13
  "tool_calls": null
14
}

If the call text lands in content, stop tuning your agent. No prompt change, schema change, or framework setting will fix it, the endpoint has to change.

Fixing it#

Once you've confirmed a serving-side gap, you have three real options:

Use a provider that serves the model correctly. For the models on LiveKit Inference, the tool-call and reasoning parsers run at the serving layer, with schema-adherence rough edges corrected there too, so tool_calls arrive well-formed before the response ever reaches your agent. That's why the same Gemma 4 31B that fails behind a generic compat layer scores at the top of LiveKit's task-completion evals.

Self-host with the parsers configured. vLLM and SGLang both support per-model tool-call parsers; you own the flags. Verify with the curl test above after any model or server upgrade, because a parser mismatch fails silently.

Prefer the provider's native API when its OpenAI-compat layer is incomplete. Compat layers are often the weaker path, less complete coverage, slower to pick up new model formats. On Bedrock, the native Converse API returns structured toolUse blocks for supported models, and LiveKit Agents has a plugin for it. Newer OpenAI-compat endpoints exist too; still run the curl test before you trust them.

The takeaway#

Model capability and endpoint parsing are separate axes, and benchmarks only tell you about the first one. Before you write off an open model, or an agent framework, spend the sixty seconds on the curl test. When an agent starts reading its tool calls out loud, the model usually made the right call. The serving stack just dropped it on the floor.

Continue reading about Latency Optimized Inference: Gemma 4 on LiveKit — how we serve Gemma 4 31B for voice agents, including the serving-layer tool-call handling discussed in this post.