Voice Activity Detection: Knowing When the Caller Is Done
How voice activity detection tells speech from silence on a call, why timing goes wrong, and how to tune the silence wait so an agent does not interrupt or lag.
When two people talk on the phone they rarely think about who speaks next. A voice agent has to calculate it. The first step is a surprisingly simple question asked many times a second: is someone speaking right now? The component that answers is called voice activity detection, or VAD.

This guide explains what VAD does, why it is harder on a phone line than in a quiet room, what the settings mean and how to tune them. For the bigger picture of how a voice agent works, read How Voice AI Works on a Phone Call.
What voice activity detection does
VAD looks at a stream of audio in very short slices and labels each one as speech or non-speech. Simple versions only measure loudness. Better ones are trained models that recognise the pattern of human speech and ignore keyboard clicks, a passing lorry or a television in the background.
The labels feed three decisions in a voice agent:
- Start of speech: the caller has begun talking, so the agent should listen and, if it is speaking, consider stopping.
- End of speech: the caller has gone quiet long enough that their turn is probably over, so the agent can answer.
- Nothing to transcribe: if the audio is only noise, the speech recogniser is not asked to guess words.
Because everything after VAD depends on it, a poor VAD makes a good agent look bad. A reply that arrives while the caller is still talking is rarely a language problem; it is usually a timing problem.
Why phone calls make it harder
A phone line is a narrow, compressed channel, often carrying a caller on a street, in a car or in a shop. That combination blurs the boundary between speech and noise. Why Phone Audio Is Hard for Speech Recognition looks at the audio itself; here are the VAD-specific effects.
What confuses a detector
- Background noiseTraffic, wind and café chatter can be mistaken for speech and keep the turn open.
- Quiet speakersA soft voice can be classed as silence, so the agent talks over the caller.
- Mid-sentence pausesPeople stop to think, spell a name or find a number in their bag.
- Echo and side toneThe agent's own voice leaking back can look like the caller speaking.
- Breaths and fillersSounds such as hmm or a sigh may or may not count as a turn.
The silence threshold trade-off
The setting that matters most is how long a silence must last before the system decides the caller has finished. Call it the silence threshold. It is a trade-off with no perfect value.
Short versus long silence threshold
| Short wait | Long wait | |
|---|---|---|
| Feel of the call | Quick and lively | Slower, more deliberate |
| Risk | Cuts in while the caller is thinking or reading out a number | Dead air after every answer; callers repeat themselves |
| Suits | Brief yes or no exchanges | Names, addresses, account details, older or hesitant callers |
Think about the content of the call. If you mostly ask yes or no questions, a short wait works. If callers must give an email address or a reference number, a longer wait prevents the agent from jumping in during a pause between digits.
Using different speeds for different questions
Some systems let you vary patience by question. Be quick when you ask whether the caller wants a callback; be patient when you ask for a postcode. If your platform does not support this, choose the setting that suits your hardest question, since cutting off a caller is more irritating than a short pause.
Tuning VAD settings in Call
In Call, each AI agent has a Transcriber section where you choose the language it listens for, or let it detect the language automatically. Two sliders shape how it decides the caller has finished. The silence threshold is the number of seconds of quiet before the caller's turn ends, adjustable between 0 and 1.5 seconds; lower means faster turn-taking and higher allows longer pauses. The noise sensitivity slider makes the agent more or less sensitive to background sound. A banner on that page tells you which of these settings live calls use today, so check it before you spend time adjusting.
A simple tuning loop
- Start with the defaultsMake a baseline set of test calls before changing anything.
- Test the hard casesCall from a noisy place, speak slowly, pause mid-number, speak quietly.
- Change one settingAdjust either silence or noise sensitivity, not both at once.
- Re-test and compareUse the same script so differences are caused by the setting.
- Review real callsRead transcripts for cut-off callers or long gaps and adjust again.
Consider a hypothetical caller giving a booking reference. They say the first four characters, glance at their confirmation email and continue after two seconds. With a short wait, the agent has already answered, the caller starts again and the two talk over each other. With a longer wait the same caller completes the reference in one go. Now picture a caller who only has to say yes. The longer wait costs them a second of silence on every turn. The right setting depends on which of these calls you expect most often.
Mobile callers add a further twist. Phones with aggressive noise suppression sometimes clip the first syllable of a sentence, so the start of speech arrives late. If callers often seem to begin talking and be ignored, try the more sensitive noise setting and test again from a handset.
Reading the symptoms
You can often tell which way a setting is wrong by listening to calls.
| What you hear | Likely cause | What to try |
|---|---|---|
| The agent talks over the caller | Silence wait too short, or quiet speech missed | Raise the silence threshold, increase noise sensitivity |
| Long pauses before each answer | Silence wait too long | Lower the threshold a little |
| The agent responds to nothing at all | Noise treated as speech, so the turn never closes | Reduce noise sensitivity |
| The agent ignores a soft-spoken caller | Speech treated as noise | Increase noise sensitivity |
| Replies feel slow even with a short wait | Delay elsewhere in the pipeline | See the latency guide |
Delay has several sources besides VAD. Voice AI Latency: Why Replies Feel Slow separates them, and Handling Barge-In and Interruptions covers what happens when the caller speaks while the agent is talking.
Beyond silence: smarter end-of-turn decisions
Silence alone is a blunt signal. Newer approaches also consider what was said: a sentence that ends with and, or a half-spoken number, probably means more is coming, while a complete question probably does not. These methods are described further in Turn-Taking in Voice Agents. Whatever the technique, treat vendor claims as hypotheses and verify them with your own calls.
Before you go live, confirm that
- The agent does not interrupt when callers read out numbers
- A pause of a second or two is not treated as the end of the turn
- A caller on a noisy street can still be understood
- The agent's own voice is not triggering the detector
- You have listened to calls from mobile phones, not only desk phones
- You know where to change the settings if real calls reveal a problem
What to do next in Call
Open the agent's voice settings, review the transcriber section and run a test call that rings your own phone through the line. Then open the call in the call history, where a manager can see how long each spoken turn took, and compare it with how the call felt. Small changes in the silence threshold are easier to judge with a side-by-side listen than from memory.
Frequently asked questions
What does VAD stand for?
Voice activity detection. It is the part of a voice system that tells speech apart from silence and background noise.
Is VAD the same as speech recognition?
No. VAD only decides whether someone is speaking. Speech recognition turns the speech into text. They work together, and VAD also decides when to stop listening and start replying.
Why does the agent sometimes interrupt me?
The silence wait is probably too short for the way you speak, or a pause was treated as the end of your turn. Raising the silence threshold usually helps.
Why is there a long pause before the agent answers?
The wait for silence may be long, or the delay is elsewhere, such as in generating the reply. Check the settings first, then look at response timing in the call history.
Does background noise affect the result?
Yes. Noise can be mistaken for speech and keep the turn open, and quiet speech can be mistaken for noise. Noise sensitivity settings and test calls from realistic places help you find a balance.
Can one setting suit every caller?
Rarely. Choose a value that suits your most demanding questions and review real calls, because patient and fast callers behave differently.