Voice AI technology

How to Test a Voice AI Agent Before Launch: A Practical Plan

A step-by-step plan for testing a phone voice AI agent before launch: scripts, accuracy, hearing, flow, hand-over, logging and a clear pass or fail decision.

Call editorial team · Published · 6 min read

A voice agent that answers well in a quiet office, in front of the person who built it, may behave very differently on a street corner with a caller who mumbles. Most launch problems are not mysterious: they are things nobody tried. A structured test plan finds them while they are cheap to fix.

A quality tester with a headset checking a call script on a laptop
Illustrative image

This guide gives you a practical plan: what to prepare, the six areas to test, how to judge results and what to do after launch. It ends with the tools in Call. For the metrics behind the judging, see Voice AI Evaluation Metrics.

Preparing the test

Start with the list of real calls. Gather the questions people actually ask, from emails, staff memory and call notes. Add the awkward ones: the angry caller, the vague caller, the caller who asks for something you do not offer. A good test set mixes common, rare and difficult cases.

From script to launch

  1. Write test scriptsReal questions, with the correct answer for each.
  2. Test the contentDoes the agent answer correctly and stay within bounds.
  3. Test hearingDoes it understand names, numbers and accents.
  4. Test the flowDoes the conversation move naturally and end well.
  5. Test hand-overDo requests for a person and urgent cases work.
  6. Check the recordsAre summaries, messages and logs correct.
  7. DecideCompare the results with the pass rule and choose.

Decide who tests. Include people who did not build the agent, because they will ask in unexpected ways. If your callers speak with regional accents or in several languages, include those voices too.

Testing the content

The first question is whether the agent says true things. Ask every question on your list and compare with the correct answer. Mark each response as correct, partly correct, wrong or invented. An invented answer, where the agent states something it was never given, matters most, because callers believe it.

See Voice AI Hallucinations for why invented answers occur and Retrieval for Voice Agents for how knowledge is used.

Testing what it hears

Voice agents fail often at the listening step. Names, postcodes, phone numbers and dates are the usual trouble. Test with real phones, not just a computer microphone, and include background noise, speakerphone and poor mobile reception.

CheckHowWhat to look for
NamesSpell and say unusual namesCorrect capture or a polite request to repeat
NumbersRead out a phone number and a dateCorrect digits, and a read-back
AccentsAsk colleagues with different accentsFewer misunderstandings and graceful recovery
NoiseCall from a busy placeNo false triggers, and clear repeat requests

Numbers and Names in Voice AI and Speech Recognition on Phone Audio explain these problems in more depth.

Testing the flow of conversation

Content can be right and the conversation still awkward. Listen for the opening, the pace, whether the agent talks over the caller, how it handles silence and how it ends. Interrupt it, stay silent, say "hello?" and change your mind midway.

Flow behaviours to try

  • InterruptingTalk over the agent and check that it stops and listens.
  • SilenceSay nothing and check that it prompts politely, then ends sensibly.
  • Changing your mindSwitch topic mid-answer and check that it follows.
  • EndingSay thanks and goodbye and check that the call closes cleanly.

Speed belongs here too. Note long pauses and what caused them. Voice AI Latency explains the usual reasons.

Testing hand-over and safety

Every voice agent needs a route to a person for requests it cannot or should not handle. Test it under each condition: during hours, outside hours, when nobody answers, and when the caller is distressed. Try urgent phrases. Check that the agent never claims to be human and never promises things it cannot deliver, such as a confirmed booking.

Hand-over checks

  • Asking for a person during business hours reaches the right people
  • Asking outside hours gets an honest message and a way to leave details
  • If nobody answers, the caller can leave a message
  • Urgent or distressed callers are passed on quickly
  • The agent identifies itself as an AI when asked
  • Sensitive details are not collected when they should not be

Testing records and logs

A call has two outcomes: what the caller experienced and what you record. After each test, check the summary, the transcript, the message left and any follow-up task. Are names and numbers correct? Is the reason for the call stated accurately? Does the summary invent anything? A perfect call with a wrong record still sends the team in the wrong direction. See AI Call Summaries.

Deciding pass or fail

Before you start, write the rule that decides whether the agent may launch. It can be simple: no invented answers in the test set, every hand-over case works, and a stated share of the common questions answered correctly. Having it in writing prevents a launch driven by deadline pressure.

Launch decisions

Pass rule written firstJudged afterwards
ClarityEveryone knows the barThe bar moves with the mood
PressureEasy to say not yetHard to resist a deadline
LearningFailures point to fixesFailures become arguments
RecordResults can be compared over timeNothing to compare

Keep a log of every test: date, version of the agent, caller, question, result and fix. When you change the prompt or knowledge, repeat the earlier tests, since fixes sometimes break other things.

It also pays to test the quiet failures, which no caller will complain about but which cost you over time. Check that a call which ends early still produces a sensible record, that a caller who hangs up mid-sentence does not leave a half-finished task behind, and that repeat callers are handled consistently. These cases rarely appear in a first test plan, yet they make up a surprising share of real traffic.

After launch

Testing does not stop when the agent goes live. Start with a limited scope, such as after hours or overflow, listen to a sample of real calls each day for the first week, and expand when you are comfortable. Review the questions it could not answer, then add the missing knowledge. Keep a way to roll back to a previous version if a change makes things worse. Your AI Receptionist's First Week is a good companion.

Testing in Call

Call gives you several ways to test an AI agent before callers do. The Playground under AI agents lets you chat with an agent, shows how many knowledge notes matched each answer and lists them, and lets you rate a reply or teach a better one. The agent's Test call rings your own phone through its line, so you hear it as a caller does; each such test is a real, billed call. The setup wizard also offers talking on screen, a call to your phone and having the agent call a number. A readiness check, run without dialling anyone, reports what an AI call from the line still needs.

Practice calls, under AI agents, What it learned, let AI callers phone your agents in Japanese and English. Each call is graded, and the lessons wait for your review, so nothing changes without you. Before a risky edit, save a restore point on the agent so you can roll back, and use Set live or Set to draft to control when the agent answers. See Setting Up an AI Agent in Call for the setup itself.

Frequently asked questions

How many test calls are enough?

Enough to cover every common question, every hand-over case and a sample of difficult behaviour. For a small agent that may be dozens of calls; keep going until new tests stop finding new problems.

Who should test the agent?

People who did not build it, including those with different accents and phone habits. They ask in ways the builder never thinks of.

Should I test on a real phone?

Yes. Phone audio is narrower and noisier than computer audio, and many problems only appear on a real line.

What counts as a serious failure?

An invented answer stated confidently, a hand-over that fails, or a record that sends the team the wrong way. These should block launch.

Do I need to retest after small changes?

Yes, at least a core set. Small changes to the prompt or knowledge can alter answers elsewhere.

What should I watch after going live?

Listen to a sample of calls daily at first, note questions it could not answer, and track hand-overs and complaints. Improve the knowledge each week.

Run test calls and practice calls on your AI agent in Call

In the demo console you can try incoming calls in a simulation.

Related guides