Improving Speech Recognition on Phone Calls: Practical Fixes
Why speech recognition struggles on phone lines and what helps: clearer audio, language and vocabulary settings, careful questions, read-backs and testing.
A voice agent is only as good as what it hears. If speech recognition turns "Tuesday at ten" into "two days at then", everything that follows goes wrong, however good the language model is. The same applies to call transcripts and summaries: a mistake at the recognition stage spreads into every later step.

This guide explains why phone speech is difficult, which errors are most common and which fixes are worth your time. For a background on the pipeline, see How Voice AI Works: STT, LLM and TTS, and for the audio side Why Phone Audio Is Hard for Speech Recognition.
Why phone speech is hard
A phone call squeezes a human voice through a narrow channel. High and low frequencies are cut, the signal is compressed, and packets can be lost on the way. Then add the caller's surroundings: traffic, a television, wind, a speakerphone in a kitchen. Recognition systems are trained on lots of speech, but conditions like these still push error rates up.
What degrades recognition
- Narrow bandwidthTelephone audio carries less detail than ordinary recordings, so similar sounds blur.
- Background noiseCars, crowds and machines compete with the voice.
- Poor microphonesSpeakerphones, cheap headsets and distant microphones add echo.
- Speaking styleFast speech, mumbling and unfamiliar accents are harder.
- Rare wordsNames, brands, street names and jargon the system has seldom seen.
- Overlapping speechInterruptions, side conversations and two people speaking.
Typical errors
Expect trouble with homophones (to, two, too), digits read in groups, spelled-out names, dates and times, and short words at the start of a sentence. Knowing which errors you have makes it easier to fix them. See Getting Names, Numbers and Addresses Right by Voice.
Start with the basics
Before changing anything clever, check the simple things.
A basic checklist in order
- Check the audioListen to recordings of real calls. If you cannot understand a section, neither can the system.
- Set the languageChoose the language the callers use, or enable automatic detection if callers vary.
- Tune the pausesAdjust how long silence lasts before the system decides the caller has finished.
- Tune the noise handlingMake the system more or less sensitive to background sound.
- Test with real voicesTry different speakers, devices and places.
Language and detection
If most callers speak one language, set it explicitly. Automatic detection is useful when callers vary, but it may take a few words to settle, and short utterances such as "yes" can be misjudged. For a multilingual line, read Multilingual Phone Support with AI. Accents and dialects also matter; see Accents and Dialects in Voice AI.
Pauses and turn-taking
If the system cuts people off mid-sentence, raise the silence threshold so it waits longer. If it seems slow to respond, lower it. Older callers and people reading out numbers pause more; test with them. A threshold that suits a brisk exchange may frustrate those who think aloud. See Voice Activity Detection and Turn-Taking in Voice Agents.
Noise sensitivity
A noisy street can fool the system into hearing speech where there is none, or hide quiet speech. Test in a noisy setting, not only in an office, and adjust sensitivity step by step.
Design questions that are easy to recognise
Often the cheapest improvement is not in the recogniser at all but in what the agent asks.
- Ask one thing at a time. "What is your name?" gets a cleaner answer than "Name, phone number and reason?".
- Offer a way to answer in few words, such as yes or no, or a choice of two options.
- Tell callers how to give numbers: "Please say the digits one at a time."
- Avoid long open questions where a short closed question will do.
- Give the caller time to answer before the agent speaks again.
Confirm what matters
For names, phone numbers, dates, addresses and anything with consequences, repeat what you heard and ask the caller to confirm. "I have Tuesday the fourth at ten in the morning. Is that right?" is much more reliable than assuming. Offer another way to supply difficult information, such as a message sent afterwards to the person for them to complete, where that is allowed.
Help the system with your words
Recognition errors often concentrate on words particular to your business: product names, staff names, place names, medical or technical terms. Ways to help include:
- Keep a list of the terms that are misheard and watch whether they improve.
- Where the system allows custom vocabulary or hints, add the important terms.
- Make sure your knowledge sources spell names consistently, so that later steps can match them.
- Teach a pronunciation dictionary for how the agent speaks those words aloud; that affects speech output, not recognition, but it helps callers understand and answer in kind.
Note the distinction between listening and speaking. A pronunciation dictionary changes how a word is read out, not how it is heard.
Handle failure gracefully
Even a well-tuned system mishears sometimes. Design the fallback: after one failed attempt, rephrase the question; after a second, offer another way, such as a person, a callback or a message. Never loop the caller through the same prompt three times. A graceful exit protects the relationship when the technology fails.
Measure and test
Without measurement, you cannot tell whether a change helped. Build a small test set from real calls with consent and noted permission: perhaps fifty short snippets that cover your typical voices, devices and noise. For each, compare what was said with what was recognised, and count the errors that mattered, such as a wrong digit, rather than every minor word.
A monthly recognition review
- Listen to a sample of real calls, including ones that failed.
- List the top five misheard words or phrases.
- Check whether wording, settings or vocabulary can address them.
- Re-test the same snippets after every change.
- Include noisy and accented samples, not only clean ones.
- Record changes and results so you do not repeat an experiment.
See How to Test a Voice AI Agent Before Launch and How to Evaluate a Voice Agent's Conversations.
Doing it in Call
Call lets you adjust listening for each AI agent under its voice settings. In the transcriber section you choose the language, or detect it automatically, and move a silence threshold and a noise sensitivity control. A banner on the page says which settings live calls use today. Raise the silence threshold if the agent cuts callers off mid-sentence.
- Pronunciation dictionary. Teach the AI voice how to read names, places and words it gets wrong; it changes what is spoken, not what is recognised.
- Transcripts. Read the transcript of a call under its recording, and label speakers after the call so you can see who said what.
- Call quality. Calls with an AI agent are scored after they end, with reasons and quotes, so weak calls are easy to find.
- Corrections. You can correct an answer in a call record and save the corrected answer to the agent's knowledge.
Call's recognition depends on the call and the phone line, so test on real lines. See Using Call History and Summaries in Call and Searching Call Transcripts in Call.
Frequently asked questions
Why is phone speech recognition less accurate than on my smartphone?
Phone lines carry narrower, more compressed audio than a modern microphone, and callers are often in noisy places or on speakerphone. The software has less to work with, so more errors occur.
Should I set one language or use automatic detection?
If nearly all callers use one language, set it. Use automatic detection when callers vary, and test short answers like yes and no, which can be misdetected.
How can I improve recognition of names and numbers?
Ask for one item at a time, tell callers how to say digits, confirm by reading back and add important terms to vocabulary where your system allows it.
Does a pronunciation dictionary improve what the agent hears?
No. It improves how the agent pronounces words aloud. It can help callers understand the agent, but recognition is a separate setting.
How do I know whether a change helped?
Keep a fixed set of real call snippets, count the errors that matter before and after, and re-run the same set after each change. One lucky call proves nothing.