Define success as an observable outcome
Start with the job, not the prompt. For a booking assistant, success might mean collecting the right date, checking availability, confirming the chosen time and receiving a successful booking response. A friendly goodbye is not evidence that a booking exists.
Write a small set of acceptance conditions for each call type. Include conditions that must never happen, such as confirming an appointment after the booking tool failed. Give each test an expected tool outcome and a clear end state.
Keep the assistant configuration, provider choices and test inputs with the result. Otherwise, a later change can make the old result impossible to interpret.
Build a scenario set with deliberate friction
| Scenario | What to inspect |
|---|---|
| Normal request | Correct details and successful tool result |
| Caller changes their mind | Old details are replaced before confirmation |
| Interruption | The agent stops appropriately and hears the new request |
| Silence or unclear speech | A useful recovery without invented information |
| Tool timeout or failure | No false confirmation; a clear next step |
| Human handoff | Correct routing and an understandable transition |
Add local names, accents, background noise and code-switching when they reflect your callers. Do not use sensitive production recordings as convenient test fixtures without an appropriate basis and controls.
Test in layers
Begin with controlled tool responses so you can isolate conversational decisions from an unreliable integration. Then test against the real integration in a safe environment. Finally, test the actual audio path and telephone route.
APIFleet provides browser tests, simulations and call logs. Use browser conversations for quick feedback and repeatable simulation scenarios for regression checks. Keep simulation results distinct from production evidence: a simulated caller does not reproduce every microphone, network or human behavior.
Simulation caller and judge models require your own credentials. Test usage has a cost; APIFleet does not include free credits. Budget for evaluation as part of the deployment.
Measure outcomes and listen to the failures
Track task completion, incorrect actions, repeated questions, handoff outcomes and cost per completed task. A low cost per minute can be misleading if the agent takes longer or creates work for a person afterward.
Listen to a sample of successful calls as well as failed ones. A transcript may hide awkward pauses, overlapping speech or a voice that is hard to understand. Look at tool requests and responses alongside the conversation.
If you measure latency, define the interval. Time from a caller finishing speech to the first useful response is different from model token latency. Use the same measurement and test conditions when comparing stacks.
Make the release decision before the test
Set your acceptance criteria before looking at results. Decide which failures block release, who reviews ambiguous calls and what rollback looks like. Keep a small stable regression set and add a case whenever you find a meaningful new failure.
Start with a controlled portion of traffic and observe the real outcomes. The migration guide describes a staged rollout. If you are choosing a platform, apply this same test plan to the alternatives in the comparison directory.
Continue with a practical next step
Sources and evidence
- APIFleet product and pricing · observed 2026-10-19
- APIFleet API documentation · observed 2026-10-19