Voice agent tool call latency: where the seconds go on a tool turn

Short answer: a model pass on each side of the tool

The tool turns in the table below share one shape: the model decides to call a tool, the tool runs, and the model turns the result into a reply before the voice starts. With tenants' phone booking tools, the tool was usually the biggest single part, except on Gemini's 5 September stall day. With the near-instant tools of browser tests, the model passes were.

The table gives medians in seconds for turns whose reply had to wait for a tool. gpt-live-1 is timed from the moment its voice model hands the request to its reasoning backend, the other two models from the voice-activity endpoint that closed the caller's turn, and medians of the parts do not add up to the median total.

As a share of all the waiting time in these turns, the tool took 49% on gpt-live-1 phone calls (851 tool turns, 425 calls, 11 September to 29 October 2026) and 22% on its web tests (82 tool turns, 49 calls, 11 September to 28 October 2026). On Gemini 3.1 Flash Live phone calls it took 45% on 6 September 2026 (67 tool turns, 20 calls) but only 9% once 5 September is included (153 tool turns, 59 calls), when many turns stalled before or after the tool. In every row the rest was the time before and after the tool, which holds the model passes, voice start-up and any stall, plus the occasional gap between chained tools.

On 5 September 2026, the first day APIFleet's own voice-activity detection for Gemini appears in these traces, 137 of 239 phone replies that did not wait for a tool took more than 10 s (the 239 came from 68 calls), against none of 85 such replies on 6 and 7 September (29 calls). That is why Gemini's phone rows are split; the traces do not show what caused the stalls, and the 6 September row rests on 67 tool turns in 20 calls from that one day.

What the r/AI_Agents thread asks

The thread asks how a booking agent whose tools return in a few hundred milliseconds can get first audio on tool turns under a 2.5 s P95, when the model's planning passes seem to take most of the wait. Later in the thread the asker reported about 1.9 s median and 4 s P95 on those turns, with two to four model passes per turn and a prompt of roughly 27,000 tokens. We read the thread and its replies through a public Reddit archive on 29 October 2026, so the live thread may have changed since.

Replies split four ways: trim the prompt, merge read-only tools, call tools in parallel behind a filler line, or move the flow into a server-side state machine. This page answers the question underneath, where the seconds go on a tool turn, from APIFleet's own call traces. It does not test any of those fixes.

How these calls were measured

The figures come from de-identified trace events in APIFleet's production call logs: 2,423 traced phone and web calls between 14 August and 29 October 2026. Phone means calls over Telnyx media streams, web means browser test sessions and demos, and the two are never pooled. It's observational data from booking assistants and test traffic, not a controlled benchmark.

Every time is the APIFleet server's clock at the moment the runtime saw the event, so carrier buffering and playback on the caller's phone come on top. For gpt-realtime-mini the clock starts at OpenAI's speech-stopped event, a voice-activity decision taken after its own silence wait. For Gemini 3.1 Flash Live (a preview model) it starts when APIFleet's own detector ends the caller's turn, which on these phone calls happens about 0.5 s after the last voiced audio.

gpt-live-1 reports no end of caller speech, so its tool turn starts when the voice model hands a request to a backend that then runs at least one tool, and whatever happened before that handoff is missing. Its voice usually speaks right after the handoff, so its turns end at the first audio after the backend has read the tool result; 97 of 948 phone tool turns, and 23 of 105 on web tests, are left out because the voice was still talking at that point or no backend pass or result audio followed.

For the other models a turn counts only if its first reply audio could not exist before a tool result: a tool started after the caller's endpoint, and no speech had been queued ahead of it. The parts of each turn are cut from the same timestamps, so they add up exactly to that turn's total.

Why a tool usually costs two model passes

Function calling hands control back to your application. OpenAI's Realtime guide has the client send the function output and then request a new response, GPT-Live's backend resumes only when the application returns the custom-function result and sends a new request, and Google lists Gemini 3.1 Flash Live as supporting only synchronous function calls, so the model waits for every result.

On gpt-live-1 phone calls, 843 of 948 tool turns (89%) used exactly two backend passes, and 95 (10%) used three or more because the backend chained a second tool (454 calls, 11 September to 29 October 2026). APIFleet asks that backend for one tool per pass, with parallel tool calls switched off, and on every stack it runs tool calls one at a time.

On the 851 phone tool turns split above (425 calls, same dates), the pass that chose the tool took a median 1.71 s and the pass that read its result 1.02 s, and the voice's next audio began 0.73 s after that second pass ended. Across all 3,003 delegated phone passes between 11 September and 29 October 2026, with or without tools, the median was 1.48 s on a median prompt of 17,321 input tokens (551 calls).

On gpt-realtime-mini phone calls the model took a median 0.74 s from the caller's endpoint to the tool and 0.54 s from the tool's end to the first audio (50 tool turns, 14 calls, August 2026). Plain replies in that group started 0.47 s after the endpoint (133 turns, 21 calls, 17 to 27 August 2026). Three assistants made all of these tool turns, one of them 72%, and four made the plain replies, so these figures describe those deployments rather than the model.

Prompt size could not be isolated. Web gpt-live-1 passes used smaller prompts (median 10,219 tokens; 559 passes, 109 calls, 11 September to 28 October 2026) but were not faster (median 1.60 s), and they mostly ran the GPT-5.6 Terra backend rather than the GPT-5.6 Luna backend used on phone.

How long the tools themselves took

Tool durations run from dispatch to result on the APIFleet server, so they include the network round trip to the tenant's endpoint. Tool names were stripped from the extract, so the table keeps tool calls of 20 ms or more, which are probably external HTTP tools; built-ins such as end_call finish faster than that.

The phone rows are tenants' booking and CRM endpoints, and each is concentrated: one assistant produced 62% of the gpt-live-1 phone tool calls, 94% of the Piper ones and 85% of the Deepgram ones. Web tests ran one assistant per call, likely per-visitor demos with different tools, so the phone and web rows differ in tools and deployments, not only in channel.

APIFleet's HTTP tools time out after 8 s by default, configurable per tool from 0.5 to 30 s, and the runtime hands the model an error result instead of retrying. At the 90th percentile in the table, a gpt-live-1 phone tool call already took 6.91 s.

What a one-moment filler covers, and what it misses

In APIFleet's cascaded pipeline, when the model's first pass returns a tool call without any words, the runtime speaks the tool's configured progress message, or by default 'One moment, please.' for English assistants, before the tool runs. Transfers get their own line, and booking writes stay silent because a progress line just before a booking confirmation would be merged into the confirmation the booking service checks.

That phrase can't start until the first pass has finished. On the Piper cascade on phone, one deployment of two assistants, 103 of 614 replies that didn't wait for a tool (17%; 121 calls, 25 August to 24 September 2026) opened with a progress phrase, configured or default. Its audio began a median 1.41 s after the transcript was final (p90 2.57 s; 103 turns, 82 calls, same dates), against 1.04 s for replies that began with the model's own words (511 turns, 111 calls, same dates).

gpt-live-1's voice can keep talking while the backend works, and on phone, when it was quiet at the handoff, its audio started a median 0.29 s later (934 tool turns, 450 calls, 11 September to 29 October 2026). On the 851 phone tool turns split above, it then went quiet for at least a median 5.07 s before the result audio (p90 10.65 s), and for at least a median 2.17 s on web tests (82 tool turns, 49 calls, 11 September to 28 October 2026). At least, because the runtime marks the voice as stopped only after 0.6 s without audible output.

So the filler gives the caller audio early, but the answer still waits for the tool and the second pass. A model that writes its own filler can also say it without calling the tool, which is why APIFleet's Gemini runtime re-prompts the model once when a reply was only 'One moment please' and no tool was requested.

Count a tool turn by what the reply waited for

A tool finishing before the first audio doesn't mean the audio waited for it. Of 52 phone turns on the Piper cascade where a tool completed between the caller's endpoint and the first audio (37 calls, 25 August to 20 September 2026), only 19 had a reply that depended on a tool started in that turn.

In 19 more the audio had already been requested before the tool started, 13 of them the runtime's goodbye ahead of an end_call that finishes in milliseconds, and in the other 14 the tool had started before the caller's endpoint. On the Deepgram cascade only 6 of 30 such turns qualified (22 calls, 15 to 18 August 2026), too few to split either cascade, which is why neither appears in the first table.

Pooled, those turns give tool-wait medians of 2.97 s on the Piper cascade and 2.17 s on the Deepgram cascade, figures that blend three different situations. If you instrument your own stack, tag each turn by whether its first audio was requested before or after a tool result, and report the two kinds separately.

What to try first, and what this data supports

Each step says whether it rests on these measurements, on APIFleet's code or documentation, or on vendor guidance, and its numbers repeat those above with the same samples and dates. None of these steps was tested as a controlled change on APIFleet.

1. Split the wait before optimising it (measured). Log the time to the tool, the tool itself and the time from the tool's end to first audio on every tool turn; in the turns above, the tool's share of the wait ranged from 9% to 49% by stack and day, so a total alone cannot say what to fix.
2. Time the tool where the agent calls it (measured). Phone tool calls of 20 ms or more took medians of 1.63 to 3.40 s in the tool table, network included, so check that a tool your own logs put at a few hundred milliseconds is that fast from the voice server.
3. Count the passes, then cut them (APIFleet code; the saving was not measured). Each tool costs one pass to choose it and usually one to read its result, and 95 of 948 gpt-live-1 phone tool turns (10%, 454 calls) chained a second tool and needed three or more passes. Merging read-only lookups into one tool removes a pass, and so does having the tool return the exact sentence to speak and playing it without a second model pass, which APIFleet's cascade already does for booking confirmations.
4. Treat parallel tool calls as untested here (not measured). APIFleet runs tool calls one at a time and asks the gpt-live-1 backend for one tool per pass, so this data says nothing about the gain, and parallel calls can only help when the lookups do not depend on each other.
5. Test prompt trimming instead of assuming it (vendor guidance, not measured here). OpenAI's latency guide says that, outside very large contexts, halving a prompt may improve latency by only 1 to 5 percent, while each extra request adds a round trip; our phone backend passes ran on median prompts of 17,321 tokens (3,003 passes), and prompt size was never varied on its own.
6. Plan for the silence, not only the first word (measured). The Piper cascade's progress phrase started a median 1.41 s after the transcript on phone (103 turns), and gpt-live-1's voice still left a median of at least 5.07 s of quiet before the result on phone (851 tool turns); whether a second line helps callers was not measured.
7. Set the tool timeout from your budget (APIFleet default; not measured). The default is 8 s, configurable from 0.5 to 30 s per tool; the gpt-live-1 phone tools here reached 6.91 s at the 90th percentile (1,008 tool calls), and a failed tool still costs the pass that explains the failure.
8. Read the tail for a P95 target (measured). In the first table the 90th percentile was 1.6 to 2.1 times the median, and 11 times for Gemini on phone once its stall day is included (153 tool turns).

Limits of this evidence

These are observational production and test logs from one platform, so no row is a benchmark of OpenAI, Google or any tool vendor. Rows differ in assistants, prompts, tools, tenants, dates and code revisions, and several rest on a few assistants: one made 62% of the gpt-live-1 phone tool turns, and three made all of the gpt-realtime-mini ones.

Times are server-side and leave out the carrier and playback, gpt-live-1 is timed from the handoff rather than from the caller's speech, and its audio boundaries are inferred from output samples. The 97 gpt-live-1 phone tool turns left out include those whose result arrived while the voice was still talking, so the measured turns may lean towards slower ones. Web rows are browser tests and demos, which can include automated evaluation runs, and phone rows include developer test and demo calls.

Gemini web tests are left out because 26 of their 35 tool turns (20 of 25 calls) fell on 5 September 2026, and the cascades because too few of their tool turns depended on a tool. Rates and shares from fewer than 20 calls and percentiles from fewer than 20 turns were not reported, and samples drawn from fewer than 20 calls show months rather than exact dates.

Continue with a practical next step

• Full gpt-live-1 phone metrics: openings, backend passes and cost
• Compare speech-to-speech models on latency and errors
• Gemini Live vs OpenAI Realtime on real phone calls
• Handle tool timeouts, retries and unknown results
• Connect HTTP tools and test their timeouts
• Run an A/A test before trusting a latency change
• Test failing tool calls before release
• More Reddit questions answered with call data

Frequently asked questions

Is the tool or the model the bigger part of a tool turn?

On gpt-live-1 it was the tool on phone calls with slow booking tools, and the backend passes on web tests with fast tools. On phone calls with tenants' booking and CRM tools, the tool took 49% of the summed wait (851 tool turns in 425 calls, 11 September to 29 October 2026); on web tests whose tools answered in a median 0.10 s, the backend passes before and after the tool, with the voice's start, took 74% (82 tool turns in 49 calls, 11 September to 28 October 2026).

Does a one-moment filler make tool turns faster?

It gives the caller audio sooner, but the answer still waits for the tool and the second model pass. The Piper cascade's progress phrase began a median 1.41 s after the transcript (103 phone turns in 82 calls, 25 August to 24 September 2026), and gpt-live-1's voice still went quiet for at least a median 5.07 s before the result on phone (851 tool turns in 425 calls, 11 September to 29 October 2026).

Why does a tool call usually need two model passes?

Function calling returns control to your application: the model asks for the call, you run it and send back the output, and a second model response produces the reply. OpenAI documents this for Realtime and GPT-Live delegation, and Google says Gemini 3.1 Flash Live waits for each function result. APIFleet's runtime skips the second pass when a tool ends the call, transfers it or waits silently for the caller, and its cascade also speaks a booking confirmation returned by the tool without one.

Will cutting a 27,000-token prompt fix tool-turn latency?

This data cannot say. Our gpt-live-1 phone passes ran on median prompts of 17,321 tokens and took a median 1.48 s (3,003 passes, 11 September to 29 October 2026), but prompt size was never varied on its own, and OpenAI's latency guide expects only a small gain from halving a prompt.

Are these numbers a benchmark of gpt-live-1, OpenAI Realtime or Gemini Live?

No. They are observational traces from APIFleet's booking assistants and test calls, with different assistants, tools, prompts and dates in each row, measured on the server rather than at the caller's ear.

Sources and evidence

• OpenAI documentation: Delegation and tools in GPT-Live · observed 2026-10-29
• OpenAI documentation: Realtime conversations, function calling · observed 2026-10-29
• Google Gemini API documentation: Tool use with Live API · observed 2026-10-29
• OpenAI documentation: Latency optimization · observed 2026-10-29