Voice AI technology

Why Phone Audio Is Hard for Speech Recognition (and What Helps)

Why narrowband, compressed and noisy phone audio trips up speech recognition, how to spot the symptoms and what you can fix in the setup and the script.

Call editorial team · Published · 5 min read

A voice assistant that works perfectly in a demo can stumble on a real call. The reason is rarely the language model. It is usually the audio. A telephone line is a hostile place for speech recognition, and knowing why helps you set realistic expectations and fix the problems you can fix.

A caller speaking on a mobile phone on a busy street
Illustrative image

This guide follows the sound from the caller's mouth to the recogniser and shows where quality is lost. It is a companion to Speech Recognition on Phone Calls and Voice Activity Detection.

The journey of a phone call's audio

Where the sound can be damaged

  1. MicrophoneA phone or headset captures the voice, along with the room.
  2. Handset processingNoise suppression and gain control reshape the sound.
  3. NetworkAudio is compressed and sent in packets across mobile and internet links.
  4. Telephony platformThe call is bridged, converted and sometimes mixed.
  5. RecogniserThe audio is converted to text, with whatever quality is left.

Each stage can only preserve or lose quality; none can add back what is already gone. That is why a poor first hop, such as a speakerphone in a car, limits everything after it.

Narrowband audio: the basic limit

Traditional telephone audio is narrowband: it carries a limited range of frequencies, far less than a studio microphone captures. The consequence is that high-frequency sounds, which separate letters like s, f and th, are weakened. Between s and f, or p and t, the telephone leaves little to go on. People cope by using context. Software, especially on isolated words such as a spelled name, has less context to lean on.

Some calls use wideband codecs that carry more detail, which helps. Whether a given call benefits depends on every link in the chain supporting it, and calls between different networks often fall back to the lowest common format.

Sounds that suffer most

  • Letters in spelled wordsB, D, P, T, S and F blur together, which makes spelling by voice unreliable.
  • DigitsFive and nine, or six and sixty, can be confusable without good audio.
  • Soft consonantsQuiet sounds at the end of words vanish under noise.
  • Short wordsLittle context means more mistakes, as with yes and no in a noisy setting.

Other causes of poor audio

Compression and packet loss

To save bandwidth, audio is compressed. If network packets are delayed or lost, short gaps or garbled bursts appear. A human fills in a missing syllable; a recogniser may produce a wrong word or none.

Background noise and processing

Callers phone from streets, kitchens, vehicles and shops. Noise suppression in a handset can remove noise but also clip the start of a word or flatten quiet speech. The result can be speech that sounds clean to a person yet harder for software.

Echo and speakerphone

When the caller's speaker feeds into their microphone, the agent's own voice may return as an echo, which can confuse both recognition and the detection of who is speaking. Encouraging callers to use a handset or headset when it matters helps.

Training mismatch

Many recognisers are trained largely on cleaner or wider-band recordings. Phone audio sounds different, so accuracy drops. Some providers offer models tuned for telephony; ask about them, and test.

What it looks like in a transcript

SymptomPossible audio cause
Wrong letters in spelled namesNarrowband blur between similar-sounding letters
Digits swapped or droppedPacket loss, noise or fast speech
Words missing at the startHandset noise suppression clipping the beginning
Odd words that were not saidEcho or background speech
Good accuracy on desk phones, poor on mobilesMobile noise and compression
Long gaps with no textSilence detection failing on a quiet voice

Look at the recording, not only the transcript. Listening shows whether the problem is the audio or the recognition. If you can hear it clearly and the recogniser could not, the recogniser is the weak link; if you struggle too, fix the audio.

Measuring the problem before fixing it

It is tempting to blame the recogniser for every error. A more reliable habit is to sort a sample of problem calls by cause. Take twenty calls where the agent misunderstood something. For each, listen to the recording and decide: was the audio poor, did the caller speak unusually fast or softly, was the recogniser simply wrong on clear speech, or was the misunderstanding in the language model after correct text?

Often the split is surprising. A business with many mobile callers may find most errors are audio; one with a specialised vocabulary, such as product codes, may find the recogniser is the weak point. Each cause has a different remedy, and fixing the wrong one wastes weeks.

Also look at patterns by time and route. If errors cluster at a certain hour, the cause may be network congestion on your side. If they cluster on calls from one region or carrier, the audio path may be at fault. Note the pattern before you change any setting so that you can tell afterwards whether the change worked.

Keep the sample honest by including calls that went well. If you review only failures, you cannot tell whether a change that fixes them has quietly broken calls that used to work. A before-and-after comparison on the same mixed set is the simplest safeguard.

What you can do about it

Design the conversation for poor audio

The techniques for names and numbers are explored in Getting Names, Numbers and Addresses Right by Voice.

Fix what you control

Audio and recognition in Call

In Call, each AI agent has transcriber settings: the language it listens for, which can be automatic detection, a silence threshold and a noise sensitivity slider. For staff using the browser phone, a live transcript can write down both sides of a call as it happens, labelled as you and them, and save it with the call. It needs the carrier connection and a public address, and without them the panel says why. Transcripts are kept for a limited time. Use them as a diagnostic: if the live text is poor on a particular call, play the recording and decide whether the audio or the recogniser is at fault. Browser calling itself is explained in How Browser Calling Works: WebRTC Explained.

Frequently asked questions

Why is phone audio worse than a voice message on my smartphone?

Phone calls use narrow, compressed audio designed for intelligibility to people, not for software. A voice message app usually records richer audio directly on the device.

Can speech recognition ever be perfect on the phone?

No. Even people mishear on bad lines. The aim is to reduce errors and design the conversation so mistakes are caught by confirmation.

Does a better microphone help the AI agent?

Yes for your own staff and for callers who use headsets, but you cannot control callers' equipment. Design for the worst realistic audio.

Why are spelled names so hard?

Many letters sound alike on a narrow line, and a single letter has no context to help. Offering another way to capture the detail, or reading back letters, reduces errors.

Should I tune recognition for telephony?

If your platform offers phone-tuned models or noise settings, test them on real calls. Compare results, because the right setting depends on your callers.

How can I tell if an error is audio or recognition?

Listen to the recording. If a person hears the words clearly and the transcript is wrong, suspect recognition. If the audio is poor to a person as well, fix the audio path.

Check call transcripts and audio in Call

In the demo console you can try incoming calls in a simulation.

Related guides