Voice AI technology

How to Evaluate a Voice Agent's Conversations: Metrics and Reviews

A practical way to judge a phone voice agent: the metrics worth tracking, how to sample and score calls, and how to turn findings into fixes.

Call editorial team · Published · 5 min read

Once a voice agent is answering real calls, the question changes from does it work to is it working well. Many teams answer with a vague impression from a handful of calls. A little structure gives far better information and tells you what to fix next.

A team lead reviewing call quality notes with a colleague
Illustrative image

This guide offers a simple evaluation approach for a phone voice agent: which metrics to track, how to review calls by hand, and how to turn findings into improvements. It builds on Testing a Voice AI Agent and the general measures in Call Center KPIs.

Start with what good means for you

A good call for a restaurant is not the same as a good call for a clinic. Before choosing metrics, write down in a sentence or two what a successful call looks like for each type of call you expect: the caller's question is answered, a message is taken with the right details, or the caller reaches the right person. Everything else follows from those definitions.

Five questions every evaluation asks

  • Was the caller understood?Did the agent hear the question and details correctly?
  • Was the answer right?Was it accurate, within the approved information and free of invention?
  • Did the call reach a useful end?Was the task done, a message taken or a handover made?
  • Was the experience acceptable?Pace, tone, repeats and effort for the caller.
  • Was it safe?Did the agent respect its limits and hand over when it should?

Metrics worth tracking

MetricWhat it tells youHow to get it
Task completionShare of calls where the caller's goal was metHuman review of a sample, plus call outcomes
Answer accuracyWhether statements matched approved informationHuman review against your sources
Handover rateHow often callers ask for or need a personCall results and requests in the help desk
Repeat rateHow often the agent asked the caller to say something againTranscript review
Response gapTime between the end of the caller's turn and the agent's replyPer-turn timing in call history
Abandon rateCallers who hang up before an outcomeCall log
Caller feedbackHow people felt about the callShort surveys or follow-up comments

Notice that most of the useful measures need some human judgement. Counts from a log tell you how many, but only a person reading the conversation can tell whether a call went well. Expect to do a mix of both. For measures that apply to every kind of call, see First Call Resolution and Answer Rate and Abandon Rate.

Reviewing calls by hand

A small, regular sample beats an occasional big review. Pick calls at random, plus a few that look unusual, such as very short or very long ones, calls that ended in a handover and calls that ended with the caller hanging up.

A call review routine

  1. SampleChoose a fixed number of calls each week, mixing random and unusual ones.
  2. Listen and readPlay the recording and read the transcript together.
  3. ScoreRate each call against your criteria with a simple scale.
  4. Note causesWrite down why anything went wrong.
  5. Fix and re-testChange the source, prompt or setting, then re-test the same case.

Keep scoring simple

Three levels are enough: good, acceptable and poor, or accurate, partly and wrong for individual statements. Fine-grained scales create arguments without adding insight. Have two people score a handful of the same calls occasionally; if they disagree often, your criteria need sharper wording.

Remember that transcripts can be wrong

The transcript is the recogniser's guess, not the truth. If a transcript looks odd, listen to the audio before blaming the agent. See Why Phone Audio Is Hard for Speech Recognition.

Measuring speed and rhythm

Callers notice delay more than almost any other factor. Track the gap before each reply and look at the worst cases, not only the average, since a few very long pauses spoil the impression. If the gaps are long, break down where the time goes; Voice AI Latency shows how. Pay attention to interruptions too: how often did the agent talk over the caller, and how often did it leave them waiting?

Manual review or automated scoring

Choosing a method

Manual reviewAutomated scoring
StrengthUnderstands nuance, tone and contextCovers every call quickly
WeaknessSlow; only a sampleMay miss subtle problems or misjudge
Best used forAccuracy, safety and caller experienceFlagging calls that deserve a human look

Automated scoring, including models that grade other models, is useful for triage: it can rank calls by signs of trouble so that humans look at the right ones first. It should not replace human sampling, especially for accuracy and safety. Whatever automated check you use, test it by comparing its verdicts with your own on a set of calls you have reviewed.

Reading the numbers sensibly

Small samples mislead. If you review ten calls and two went badly, you cannot say that one in five calls goes badly; you can only say that you found two problems worth fixing. Use samples to find causes, not to announce rates, unless the sample is large and random enough to support it.

Separate the agent from the situation. A drop in task completion in a week with a service outage is not a verdict on the prompt. Note events beside your metrics: a new opening hour, a campaign that sent callers with a new question, a change to the sources. A simple log of changes makes the trends explainable.

Finally, decide in advance what you will do with each result. A rise in repeats might lead you to look at the silence settings; a rise in handovers on one topic might lead you to add an exact answer for it. A metric with no linked action is just decoration.

Share results with the people who answer the phones as well as with managers. Staff often know why a pattern exists, and they are the ones who will pick up the calls the agent hands over. A short monthly review with examples, including a few calls that went well, keeps the work grounded in real conversations.

Evaluating in Call

In Call, the AI call history lists the calls of a period, with the result and, when known, the sentiment, and opens the whole conversation turn by turn. Managers can open a call to see how long each spoken turn took. The Automatic review area groups recent calls by topic, ranks recent calls with no human review by the signals on them, and offers a sample of AI summaries for you to mark as accurate, partly or wrong, which feeds a running view of summary accuracy. Conversation AI also lets you search every stored transcript for a phrase. Together these give you the counts and the sampling you need. Use the edit history to see corrections made to the agent's words.

A weekly evaluation checklist

  • Read a fixed sample of calls, random and unusual
  • Score task completion and accuracy
  • Count handovers and note why they happened
  • Check the longest response gaps
  • Search transcripts for phrases that signal confusion
  • Record each issue and the fix made
  • Re-test fixed cases before closing them

Frequently asked questions

What is the single most useful metric for a voice agent?

There is no single one. Task completion tells you whether callers got what they needed, but you also need accuracy, handover quality and timing to see why.

How many calls should I review?

A fixed weekly sample that you can sustain is better than an occasional large one. Start with a modest number and increase it after launch changes or when problems appear.

Can I rely on automated scoring alone?

Not for accuracy or safety. Use it to rank and flag calls, then have people review the flagged ones and a random sample.

Why do transcripts sometimes disagree with the recording?

The transcript is produced by speech recognition and can contain errors. Listen to the audio before judging the agent.

How do I know if a change improved things?

Keep a fixed set of test cases and a regular review routine, then compare results before and after the change using the same criteria.

Should callers be surveyed?

A short question after the call can add useful information, but keep it brief and respect any rules on contacting people. Pair it with your own review of the conversation.

Review AI conversations in your Call history

In the demo console you can try incoming calls in a simulation.

Related guides