Voice AI technology

Voice Activity Detection: Knowing When the Caller Is Done

How voice activity detection tells speech from silence on a call, why timing goes wrong, and how to tune the silence wait so an agent does not interrupt or lag.

Call editorial team · Published · 6 min read

When two people talk on the phone they rarely think about who speaks next. A voice agent has to calculate it. The first step is a surprisingly simple question asked many times a second: is someone speaking right now? The component that answers is called voice activity detection, or VAD.

A person speaking into a phone headset at a desk
Illustrative image

This guide explains what VAD does, why it is harder on a phone line than in a quiet room, what the settings mean and how to tune them. For the bigger picture of how a voice agent works, read How Voice AI Works on a Phone Call.

What voice activity detection does

VAD looks at a stream of audio in very short slices and labels each one as speech or non-speech. Simple versions only measure loudness. Better ones are trained models that recognise the pattern of human speech and ignore keyboard clicks, a passing lorry or a television in the background.

The labels feed three decisions in a voice agent:

Because everything after VAD depends on it, a poor VAD makes a good agent look bad. A reply that arrives while the caller is still talking is rarely a language problem; it is usually a timing problem.

Why phone calls make it harder

A phone line is a narrow, compressed channel, often carrying a caller on a street, in a car or in a shop. That combination blurs the boundary between speech and noise. Why Phone Audio Is Hard for Speech Recognition looks at the audio itself; here are the VAD-specific effects.

What confuses a detector

  • Background noiseTraffic, wind and café chatter can be mistaken for speech and keep the turn open.
  • Quiet speakersA soft voice can be classed as silence, so the agent talks over the caller.
  • Mid-sentence pausesPeople stop to think, spell a name or find a number in their bag.
  • Echo and side toneThe agent's own voice leaking back can look like the caller speaking.
  • Breaths and fillersSounds such as hmm or a sigh may or may not count as a turn.

The silence threshold trade-off

The setting that matters most is how long a silence must last before the system decides the caller has finished. Call it the silence threshold. It is a trade-off with no perfect value.

Short versus long silence threshold

Short waitLong wait
Feel of the callQuick and livelySlower, more deliberate
RiskCuts in while the caller is thinking or reading out a numberDead air after every answer; callers repeat themselves
SuitsBrief yes or no exchangesNames, addresses, account details, older or hesitant callers

Think about the content of the call. If you mostly ask yes or no questions, a short wait works. If callers must give an email address or a reference number, a longer wait prevents the agent from jumping in during a pause between digits.

Using different speeds for different questions

Some systems let you vary patience by question. Be quick when you ask whether the caller wants a callback; be patient when you ask for a postcode. If your platform does not support this, choose the setting that suits your hardest question, since cutting off a caller is more irritating than a short pause.

Tuning VAD settings in Call

In Call, each AI agent has a Transcriber section where you choose the language it listens for, or let it detect the language automatically. Two sliders shape how it decides the caller has finished. The silence threshold is the number of seconds of quiet before the caller's turn ends, adjustable between 0 and 1.5 seconds; lower means faster turn-taking and higher allows longer pauses. The noise sensitivity slider makes the agent more or less sensitive to background sound. A banner on that page tells you which of these settings live calls use today, so check it before you spend time adjusting.

A simple tuning loop

  1. Start with the defaultsMake a baseline set of test calls before changing anything.
  2. Test the hard casesCall from a noisy place, speak slowly, pause mid-number, speak quietly.
  3. Change one settingAdjust either silence or noise sensitivity, not both at once.
  4. Re-test and compareUse the same script so differences are caused by the setting.
  5. Review real callsRead transcripts for cut-off callers or long gaps and adjust again.

Consider a hypothetical caller giving a booking reference. They say the first four characters, glance at their confirmation email and continue after two seconds. With a short wait, the agent has already answered, the caller starts again and the two talk over each other. With a longer wait the same caller completes the reference in one go. Now picture a caller who only has to say yes. The longer wait costs them a second of silence on every turn. The right setting depends on which of these calls you expect most often.

Mobile callers add a further twist. Phones with aggressive noise suppression sometimes clip the first syllable of a sentence, so the start of speech arrives late. If callers often seem to begin talking and be ignored, try the more sensitive noise setting and test again from a handset.

Reading the symptoms

You can often tell which way a setting is wrong by listening to calls.

What you hearLikely causeWhat to try
The agent talks over the callerSilence wait too short, or quiet speech missedRaise the silence threshold, increase noise sensitivity
Long pauses before each answerSilence wait too longLower the threshold a little
The agent responds to nothing at allNoise treated as speech, so the turn never closesReduce noise sensitivity
The agent ignores a soft-spoken callerSpeech treated as noiseIncrease noise sensitivity
Replies feel slow even with a short waitDelay elsewhere in the pipelineSee the latency guide

Delay has several sources besides VAD. Voice AI Latency: Why Replies Feel Slow separates them, and Handling Barge-In and Interruptions covers what happens when the caller speaks while the agent is talking.

Beyond silence: smarter end-of-turn decisions

Silence alone is a blunt signal. Newer approaches also consider what was said: a sentence that ends with and, or a half-spoken number, probably means more is coming, while a complete question probably does not. These methods are described further in Turn-Taking in Voice Agents. Whatever the technique, treat vendor claims as hypotheses and verify them with your own calls.

Before you go live, confirm that

  • The agent does not interrupt when callers read out numbers
  • A pause of a second or two is not treated as the end of the turn
  • A caller on a noisy street can still be understood
  • The agent's own voice is not triggering the detector
  • You have listened to calls from mobile phones, not only desk phones
  • You know where to change the settings if real calls reveal a problem

What to do next in Call

Open the agent's voice settings, review the transcriber section and run a test call that rings your own phone through the line. Then open the call in the call history, where a manager can see how long each spoken turn took, and compare it with how the call felt. Small changes in the silence threshold are easier to judge with a side-by-side listen than from memory.

Frequently asked questions

What does VAD stand for?

Voice activity detection. It is the part of a voice system that tells speech apart from silence and background noise.

Is VAD the same as speech recognition?

No. VAD only decides whether someone is speaking. Speech recognition turns the speech into text. They work together, and VAD also decides when to stop listening and start replying.

Why does the agent sometimes interrupt me?

The silence wait is probably too short for the way you speak, or a pause was treated as the end of your turn. Raising the silence threshold usually helps.

Why is there a long pause before the agent answers?

The wait for silence may be long, or the delay is elsewhere, such as in generating the reply. Check the settings first, then look at response timing in the call history.

Does background noise affect the result?

Yes. Noise can be mistaken for speech and keep the turn open, and quiet speech can be mistaken for noise. Noise sensitivity settings and test calls from realistic places help you find a balance.

Can one setting suit every caller?

Rarely. Choose a value that suits your most demanding questions and review real calls, because patient and fast callers behave differently.

Tune how your AI agent decides a caller has finished in Call

In the demo console you can try incoming calls in a simulation.

Related guides