Where the data comes from
Every measurement in this note comes from APIFleet's internal evaluation harness, run on 12 and 13 August 2026. All callers were simulated: no real caller audio, production call or customer data was used. The run records stay internal because they hold prompts, transcripts and synthetic audio, so this note reports aggregates and method rather than linking them.
In the A/A test and the runs around it, a language model (GPT-4.1 mini, sampling at temperature 0.9) played each caller from a persona and a goal with seeded booking details, and gpt-4o-mini-tts voiced its lines. The harness streamed that audio at real-time pace in 20 ms frames to a restaurant-booking test assistant over APIFleet's test-call WebSocket, adding the pauses, line noise and interruptions each persona called for.
In the A/A test, calls allowed up to four caller turns and lasted about 46 seconds on average. Each call received a 0 to 100 naturalness composite: deterministic timing, turn-taking, interruption, repetition and speaking-rate measures, plus 0 to 10 ratings from three OpenAI transcript judges (GPT-4.1 mini, GPT-4.1 and GPT-4o, with the median taken per criterion) and one audio judge, gpt-audio-mini, that listened to the assistant's track.
Task correctness was scored separately and never folded into naturalness: code compared the name, phone number, party size, day, time and requested action with the scenario's expected values. In the A/A test it matched what the assistant said aloud; from the 30-call runs onward, a GPT-4.1 mini extractor first reported the booking as the assistant understood it, so a detail captured correctly but never repeated back no longer counted as a miss.
| Simulated caller | Scenario |
|---|---|
| Normal caller | Restaurant reservation |
| Goes quiet for a long time | Two requests in a single breath |
| Provides incomplete information | Ambiguous date |
| Calling from a noisy street | Change the number of guests |
| Confused caller | Goes silent mid-sentence, then resumes |
| Slow speaker | Talks over the greeting |
| Northern Irish accent | Asks to be transferred |
| Corrects themselves | Cancel a reservation |
| Poor telephone connection | Asks for something to be repeated |
| Speaks over the receptionist | Complicated booking |
What two identical configurations scored
The A/A test used two separately provisioned copies of one assistant, checked field by field to be identical, and gave both the same ten caller, scenario and seed combinations. The arms ran at the same time, each placing its calls in sequence, so vendor conditions at that hour affected both. Any difference between them is noise by construction.
Our own leaderboard ranked the arms by naturalness and named twin B the winner. In the clean runs of the same batch, the voice-speed and prompt changes we were testing measured between about 1 and 10 points of naturalness, so noise of this size is not a rounding error.
| Metric | Twin A | Twin B | Gap |
|---|---|---|---|
| Naturalness composite (0 to 100) | 50.20 | 53.11 | 2.91 |
| Response timing component | 22.0 | 32.1 | 10.1 |
| Human-likeness component | 45.0 | 50.0 | 5.0 |
| Conversational language component | 56.0 | 61.0 | 5.0 |
| Task correctness | 55.7% | 50.7% | 5.0 points |
Why ten calls per arm cannot settle small changes
Arm averages hide how much single calls vary. Computed from the stored per-call scores, naturalness had a standard deviation of about 7 points in one twin and 5 in the other, which puts the standard error of a difference between two 10-call averages near 2.8 points. The 2.91-point gap was therefore about one standard error, a typical size for pure noise.
Matching calls by caller, scenario and seed did not make the twins move together: the per-call correlation between arms was -0.37, and a paired bootstrap gives a 95% interval of -3.1 to +8.8 points for the gap. Likely reasons: the simulated caller samples its lines at temperature 0.9, and the seed fixed only the booking details, not the random draws behind the pauses, line noise and interruptions, so the same seed does not produce the same conversation.
Task correctness was noisier still. With one to five checks per call, per-call rates ranged from 0% to 100% with a standard deviation of about 36 points, so a 10-call difference has a standard error near 16 points (9 when paired by scenario), and the twins' 5-point gap understates the noise rather than bounding it.
A later version of the harness replayed byte-identical caller audio in both arms and collected six judgments per call. Its 10-call paired comparisons still produced 95% intervals about 5 to 7 points wide for the naturalness difference.
How many calls per arm a 3-point change needs
Our first estimate scaled each A/A gap by the square root of the sample size and asked when it would shrink to 3 points. That rule suggested about 10 calls per arm for naturalness, 28 for human-likeness and task correctness, and 114 for response timing.
Treat those figures as floors. At those sample sizes a 3-point change is only as large as typical noise, and the rule rests on a single A/A draw.
A conventional power calculation uses the per-call standard deviation instead. At 5% two-sided significance and 80% power, calls per arm come to about 15.7 × (standard deviation ÷ smallest change worth detecting)² for two independent arms, which gives the right-hand column below. That is the textbook normal-approximation formula for a single mean, about 7.85 × (standard deviation ÷ change)², doubled because a difference between two arm averages carries both arms' noise.
For illustration only: a metric with a 10-point per-call standard deviation and a 5-point change of interest needs about 15.7 × 4, or 63, calls per arm. When both arms share scenarios, analyze the paired differences instead; in our runs that helped task correctness most, because a scenario that defeated one arm usually defeated the other.
Both columns are rough estimates. Each standard deviation comes from only 19 or 20 calls, and under a normal approximation its 95% uncertainty alone allows the required sample size to be about 0.6 to 2.2 times the figure shown, or roughly 40 to 150 calls per arm for naturalness.
| Metric | A/A gap at 10 calls | Per-call standard deviation | Gap-scaling floor | Power-based estimate |
|---|---|---|---|---|
| Naturalness composite | 2.91 | 6.2 | about 10 | about 67 |
| Human-likeness | 5.0 | 9.6 | about 28 | about 160 |
| Response timing | 10.1 | 20.8 | about 114 | about 760 |
| Task correctness | 5.0 points | 35.7 points | about 28 | about 2,200 (about 700 paired) |
One prompt change, four different answers
Around the A/A test, we measured one prompt variant four times. The run notes describe it as banning call-centre boilerplate and repeated confirmation, later notes add short replies, and the run records keep only these notes rather than the prompt text, so its wording may have varied slightly between runs. Its measured effect on naturalness changed sign three times, and a fifth attempt, in which 7 of 12 calls per arm failed, produced no usable reading.
The +10.2 felt like a discovery and was the reading most likely to be luck: 8 calls per arm, no judges, and the best of three candidate arms. Even a paired bootstrap on that run excluded zero (+2.5 to +18.6 points), a reminder that intervals from tiny samples, read after picking the winner, are optimistic.
Rerun with 30 calls per arm on a new seed, the variant was indistinguishable from the control on naturalness (61.36 against 60.29, paired interval -2.7 to +5.3) and on task correctness (52.2% against 52.9%). That is consistent with the +10.2 having been noise, although the interval still allows a real effect of up to about 5 points.
| Reading | Calls per arm | Scoring | Naturalness change |
|---|---|---|---|
| First run, later found contaminated by a connection keepalive bug | 8 (7 scored) | Deterministic metrics only | -5.7 |
| Clean rerun | 8 | Deterministic metrics only | +10.2 |
| Rerun on a new seed | 10 | With judges | -3.6 |
| Larger rerun on a new seed | 30 | Deterministic metrics only | +1.1 |
The judges disagreed with each other as well
The harness flags a call for human review when the three transcript judges' ratings for any criterion differ by 3 or more points on the 0 to 10 scale. In the A/A test, 14 of the 20 calls were flagged, and the widest spread was 8 points.
Spreads of 3 points or more clustered on conciseness, repetition and filler use, and were rare for turn-taking and context awareness. In a later run, one panel split 8, 1 and 1 on fillers and backchannels: one judge rewarded the absence of awkward fillers, while two penalized the absence of any fillers.
A median of three contains one outlier, but it cannot repair a criterion that judges read in different ways. Calibrate ambiguous criteria against human ratings before letting them gate a release, and keep exact checks such as names and phone numbers out of the judges' hands.
The simulated caller is part of the instrument
On 13 August 2026, one step of a diagnostic sequence on a dental-receptionist test assistant changed only the simulated caller's voice, from the operating system's built-in speech synthesizer to a neural streaming voice. Calls with an exactly correct phone number rose from 3 of 12 to 12 of 12, and task correctness from 65.8% to 85.4%, without any change to the assistant.
With the built-in voice, transcription had repeatedly dropped the leading zero from Irish phone numbers, so the test was partly scoring its own audio. Neural caller voices are stochastic, so the harness now caches the control arm's caller audio, replays identical bytes in the candidate arm and records a SHA-256 hash of every utterance so the match can be checked.
In one later voice comparison, all 39 caller utterances common to both arms matched byte for byte. If your test caller is a synthetic voice, test the caller too, because a digit your transcriber mishears in its audio is scored as your agent's mistake.
What a change that beats the noise looks like
On the night of the A/A test, we compared two otherwise identical assistants that differed only in a 47-term list of Irish first names, surnames and Dublin areas sent to Deepgram Nova-3 as keyterm prompts. Both arms reused the 30 caller, scenario and seed combinations of an earlier run in which 35 of 42 name checks (83%) had failed; the recorded examples were Irish names transcribed as phonetic near-misses or missed entirely.
Name accuracy rose from 3 of 21 checks to 13 of 21. Eleven calls flipped from wrong to right and one the other way, which an exact McNemar test puts at p ≈ 0.006, and task correctness rose from 60.1% to 77.2% with a paired bootstrap 95% interval of +8.0 to +27.1 points.
Naturalness showed no clear change (58.72 with keyterms against 60.27 without, paired interval -6.5 to +2.9, scored without judges), which fits: keyterms change what the assistant hears, not how it speaks. Our leaderboard ranked arms by naturalness and still named the arm without keyterms the winner, so the sweep now reports a correctness lead separately.
A later grader fix stopped penalizing the assistant when the simulated caller never said the detail being checked, for example by hanging up before giving a name. Re-scoring the stored calls gave 67.8% without keyterms against 81.7% with them, a gap of about 14 points with a paired interval of +3.9 to +24.4, while the A/A twins re-scored 5.8 points apart.
Deepgram's documentation describes keyterm prompting for Nova-3 and Flux, passed as repeated keyterm parameters, and suggests focusing on roughly 20 to 50 important terms. In APIFleet's cascading pipeline, an assistant's speech keyterms setting is passed to Nova-3 the same way, capped at 100 terms.
Harness bugs that produced plausible numbers
Each of these bugs produced a believable score rather than an error, which is why each survived until someone checked. Treat the evaluation harness as code under test: assert coverage, fail loudly when a judge is missing, and refuse to rank arms whose calls failed.
| Bug | What the harness reported |
|---|---|
| Audio judge configured with a retired model | Normal-looking scores with the 15% audio component silently missing, for three evaluation runs |
| Scenario sampler that stepped by total divided by limit | Full coverage, while only 5 of 22 scenarios were tested |
| Leaderboard that ranked arms with failed calls | A winner chosen from 32 calls that never connected |
| Speaking rate measured on audio cut short by barge-in | 307 words per minute |
| Acceptance threshold of 2.0 points | Noise accepted as improvement twice, below a 2.91-point spread |
A checklist for your own voice agent evaluation
These are the steps we now follow before trusting a comparison, whether the calls come from APIFleet simulations, another platform or a custom harness. In APIFleet, running the same simulation repeatedly against an unchanged assistant gives you the A/A spread for your own personas, scenarios and judge checks.
- Run an A/A test first: two identical configurations, the same scenarios and seeds, and the same judges and rubrics you plan to use for decisions.
- Record the per-call standard deviation of every metric, not just arm averages, and repeat the A/A test whenever judges, rubrics, the caller or the pipeline change.
- Size each comparison from that spread, using about 15.7 × (standard deviation ÷ smallest change worth detecting)² calls per arm for 5% two-sided significance and 80% power.
- Set acceptance thresholds above the measured noise; our harness now uses roughly twice the A/A spread observed at 10 calls per arm.
- Keep task correctness as a gate rather than a score term, and check names, numbers and tool arguments with deterministic comparisons.
- Run both arms at the same time, replay byte-identical caller audio in voice comparisons, and verify how the synthetic caller pronounces names and digits.
- Report a paired interval with every decision, and re-score stored calls whenever the grader changes so old and new numbers stay comparable.
- Change one thing per experiment, keep production unchanged until a candidate clears the gates, and keep rejected runs on record.
Limits of this evidence
These are small internal experiments on one harness, two test businesses (a restaurant and a dental practice) and simulated callers. A single A/A pair is one draw of the noise, and the spread depends on your models, prompts, judges, caller and network, so re-measure it when any of them changes.
Simulated callers do not reproduce every human behavior, microphone or phone line. Use results like these to design tests, not to replace reviewing real calls where you have an appropriate basis to do so.
None of these runs changed a production configuration, and none is a benchmark of the vendors or models named here.
Continue with a practical next step
Frequently asked questions
What is an A/A test for a voice agent?
Two identical configurations run through the same scenarios with the same scoring. Because nothing differs, every gap between them is noise, which tells you how large a real change must be before your harness can see it.
How many test calls do I need to compare two voice agent configurations?
Measure the per-call standard deviation of your metric first. In our harness, resolving a 3-point naturalness change at 5% significance and 80% power would need roughly 67 calls per arm (plausibly 40 to 150, given how few calls the estimate rests on), while a large change such as the keyterm result was clear at 30 paired calls.
Can LLM judges decide whether a voice agent is ready to release?
Use them for perceptual qualities and keep release gates deterministic where you can. In our A/A test, the three transcript judges differed by 3 or more points on at least one criterion in 14 of 20 calls.
Did this note use real customer calls?
No. Every caller was simulated by a language model and a text-to-speech voice, and no production call, real caller audio or customer data was used.
Sources and evidence
- Deepgram documentation: Keyterm Prompting · observed 2026-10-29
- NIST/SEMATECH e-Handbook of Statistical Methods: sample sizes required · observed 2026-10-29
- APIFleet documentation: simulations, personas, scenarios and checks · observed 2026-10-29