Why Phone Audio Is Hard for Speech Recognition (and What Helps)
Why narrowband, compressed and noisy phone audio trips up speech recognition, how to spot the symptoms and what you can fix in the setup and the script.
A voice assistant that works perfectly in a demo can stumble on a real call. The reason is rarely the language model. It is usually the audio. A telephone line is a hostile place for speech recognition, and knowing why helps you set realistic expectations and fix the problems you can fix.

This guide follows the sound from the caller's mouth to the recogniser and shows where quality is lost. It is a companion to Speech Recognition on Phone Calls and Voice Activity Detection.
The journey of a phone call's audio
Where the sound can be damaged
- MicrophoneA phone or headset captures the voice, along with the room.
- Handset processingNoise suppression and gain control reshape the sound.
- NetworkAudio is compressed and sent in packets across mobile and internet links.
- Telephony platformThe call is bridged, converted and sometimes mixed.
- RecogniserThe audio is converted to text, with whatever quality is left.
Each stage can only preserve or lose quality; none can add back what is already gone. That is why a poor first hop, such as a speakerphone in a car, limits everything after it.
Narrowband audio: the basic limit
Traditional telephone audio is narrowband: it carries a limited range of frequencies, far less than a studio microphone captures. The consequence is that high-frequency sounds, which separate letters like s, f and th, are weakened. Between s and f, or p and t, the telephone leaves little to go on. People cope by using context. Software, especially on isolated words such as a spelled name, has less context to lean on.
Some calls use wideband codecs that carry more detail, which helps. Whether a given call benefits depends on every link in the chain supporting it, and calls between different networks often fall back to the lowest common format.
Sounds that suffer most
- Letters in spelled wordsB, D, P, T, S and F blur together, which makes spelling by voice unreliable.
- DigitsFive and nine, or six and sixty, can be confusable without good audio.
- Soft consonantsQuiet sounds at the end of words vanish under noise.
- Short wordsLittle context means more mistakes, as with yes and no in a noisy setting.
Other causes of poor audio
Compression and packet loss
To save bandwidth, audio is compressed. If network packets are delayed or lost, short gaps or garbled bursts appear. A human fills in a missing syllable; a recogniser may produce a wrong word or none.
Background noise and processing
Callers phone from streets, kitchens, vehicles and shops. Noise suppression in a handset can remove noise but also clip the start of a word or flatten quiet speech. The result can be speech that sounds clean to a person yet harder for software.
Echo and speakerphone
When the caller's speaker feeds into their microphone, the agent's own voice may return as an echo, which can confuse both recognition and the detection of who is speaking. Encouraging callers to use a handset or headset when it matters helps.
Training mismatch
Many recognisers are trained largely on cleaner or wider-band recordings. Phone audio sounds different, so accuracy drops. Some providers offer models tuned for telephony; ask about them, and test.
What it looks like in a transcript
| Symptom | Possible audio cause |
|---|---|
| Wrong letters in spelled names | Narrowband blur between similar-sounding letters |
| Digits swapped or dropped | Packet loss, noise or fast speech |
| Words missing at the start | Handset noise suppression clipping the beginning |
| Odd words that were not said | Echo or background speech |
| Good accuracy on desk phones, poor on mobiles | Mobile noise and compression |
| Long gaps with no text | Silence detection failing on a quiet voice |
Look at the recording, not only the transcript. Listening shows whether the problem is the audio or the recognition. If you can hear it clearly and the recogniser could not, the recogniser is the weak link; if you struggle too, fix the audio.
Measuring the problem before fixing it
It is tempting to blame the recogniser for every error. A more reliable habit is to sort a sample of problem calls by cause. Take twenty calls where the agent misunderstood something. For each, listen to the recording and decide: was the audio poor, did the caller speak unusually fast or softly, was the recogniser simply wrong on clear speech, or was the misunderstanding in the language model after correct text?
Often the split is surprising. A business with many mobile callers may find most errors are audio; one with a specialised vocabulary, such as product codes, may find the recogniser is the weak point. Each cause has a different remedy, and fixing the wrong one wastes weeks.
Also look at patterns by time and route. If errors cluster at a certain hour, the cause may be network congestion on your side. If they cluster on calls from one region or carrier, the audio path may be at fault. Note the pattern before you change any setting so that you can tell afterwards whether the change worked.
Keep the sample honest by including calls that went well. If you review only failures, you cannot tell whether a change that fixes them has quietly broken calls that used to work. A before-and-after comparison on the same mixed set is the simplest safeguard.
What you can do about it
Design the conversation for poor audio
- Confirm key details. Read back numbers and names in short chunks and let the caller correct them.
- Avoid spelling out loud where you can. Offer alternatives such as a reference number, or take details by message or booking page.
- Use short, direct questions. Fewer words mean fewer chances to mishear.
- Offer a person. If the agent fails twice to understand, hand over rather than loop.
The techniques for names and numbers are explored in Getting Names, Numbers and Addresses Right by Voice.
Fix what you control
- Use quality headsets for your own staff, since their side of the call also matters for transcripts.
- Check your network for jitter and packet loss, especially on shared office internet.
- Choose recognition settings suited to phone audio, and adjust noise sensitivity if your platform offers it.
- Test from mobile phones and noisy places.
Audio and recognition in Call
In Call, each AI agent has transcriber settings: the language it listens for, which can be automatic detection, a silence threshold and a noise sensitivity slider. For staff using the browser phone, a live transcript can write down both sides of a call as it happens, labelled as you and them, and save it with the call. It needs the carrier connection and a public address, and without them the panel says why. Transcripts are kept for a limited time. Use them as a diagnostic: if the live text is poor on a particular call, play the recording and decide whether the audio or the recogniser is at fault. Browser calling itself is explained in How Browser Calling Works: WebRTC Explained.
Frequently asked questions
Why is phone audio worse than a voice message on my smartphone?
Phone calls use narrow, compressed audio designed for intelligibility to people, not for software. A voice message app usually records richer audio directly on the device.
Can speech recognition ever be perfect on the phone?
No. Even people mishear on bad lines. The aim is to reduce errors and design the conversation so mistakes are caught by confirmation.
Does a better microphone help the AI agent?
Yes for your own staff and for callers who use headsets, but you cannot control callers' equipment. Design for the worst realistic audio.
Why are spelled names so hard?
Many letters sound alike on a narrow line, and a single letter has no context to help. Offering another way to capture the detail, or reading back letters, reduces errors.
Should I tune recognition for telephony?
If your platform offers phone-tuned models or noise settings, test them on real calls. Compare results, because the right setting depends on your callers.
How can I tell if an error is audio or recognition?
Listen to the recording. If a person hears the words clearly and the transcript is wrong, suspect recognition. If the audio is poor to a person as well, fix the audio path.