Voice AI technology

Speech-to-Speech Models vs Cascaded Voice Pipelines Compared

Two ways to build a phone voice agent: speech-to-speech models and cascaded pipelines. What each does well, where each is weak and how to choose by testing.

Call editorial team · Published · 5 min read

When you hear a voice agent speak, one of two designs is usually behind it. In the first, software turns the caller's speech into text, a language model writes a text reply, and a voice engine reads it aloud. In the second, a single model listens to audio and answers in audio. Vendors argue about which is better. The honest answer is that each trades something.

Engineers comparing audio waveforms on a laptop screen
Illustrative image

This guide compares the two in plain language so you can ask better questions when choosing a platform. For a wider introduction, start with How Voice AI Works on a Phone Call.

The cascaded pipeline

A cascaded pipeline, sometimes called speech-to-text, then model, then text-to-speech, has been the standard approach for years. Each stage is a separate component, and text passes between them.

Cascaded pipeline

  1. Speech recognitionConverts the caller's audio into text.
  2. Language modelReads the text, with instructions and sources, and writes a reply.
  3. Speech synthesisTurns the reply text into spoken audio.
  4. TelephonyCarries the audio to and from the caller.

Because the middle of the system is plain text, you can read it, search it, store it, check it against rules and replace any stage. You can swap in a recogniser that is better with your callers' accents or a voice that suits your brand.

The speech-to-speech approach

A speech-to-speech model, also described as an end-to-end or native audio model, handles audio directly. It listens to the sound itself and generates sound as output, though many such systems also produce text for logging.

Speech-to-speech

  1. Audio inThe caller's voice goes straight to the model.
  2. One modelIt interprets meaning, tone and timing and plans a reply.
  3. Audio outIt generates the spoken response directly.
  4. TelephonyThe audio travels to and from the caller.

The appeal is that nothing is lost between stages. A pause, a laugh, hesitation or a sigh can influence the reply, and the model can start speaking sooner because it is not waiting for three separate steps.

Side by side

Cascaded versus speech-to-speech

CascadedSpeech-to-speech
NaturalnessDepends on each stage; can feel stepwiseOften smoother, with timing and tone preserved
Response timeSum of three stages, which can be tunedPotentially shorter, since fewer hand-offs
ControlHigh: rules can be applied to text at each stepLower: harder to inspect what happens inside
Transcripts and logsNatural by-productUsually generated separately and may differ from what was said
Swapping partsEasy: change recogniser or voice aloneLimited to what the model offers
Voice choiceWide choice of voicesOften a smaller set
MaturityLong track recordNewer, changing quickly

Treat any claim of absolute superiority with caution. Quality varies between products inside each category, and results on a clean demo may not hold on a noisy mobile call.

What matters for phone calls

Phone audio is narrow and noisy

Telephone audio is compressed and often noisy. Whichever design you choose, check how it performs on real phone audio and not just on studio recordings. Why Phone Audio Is Hard for Speech Recognition explains the effect.

Control and auditability

If you must demonstrate what an agent said and why, text at each stage helps. A cascaded design lets you apply a check between the model's reply and the voice. With an end-to-end model you rely more on the vendor's controls and on testing. Review Guardrails for Voice AI Agents and decide what you need to enforce.

Interruptions and timing

Native audio models often handle overlap and interruptions more fluidly because they hear the caller continuously. Cascaded systems rely on detection of speech and silence to decide when to stop and start; see Voice Activity Detection and Turn-Taking in Voice Agents. Better engineering can narrow the gap, and the gap may keep changing.

Languages and accents

Check the languages and accents that matter to you by testing, rather than accepting a list in a brochure. A model strong in one language may be weaker in another, and the same holds for individual stages in a pipeline.

Choosing for your situation

A few hypothetical scenarios show how priorities pull in different directions.

Questions to ask a vendor

  • Which stages does the system use, and can I change them?
  • Do I get a transcript of every call, and is it what the model heard?
  • How are rules and limits enforced?
  • How does it behave on mobile and noisy lines?
  • What are typical response times on real phone calls?
  • How are interruptions handled?
  • What happens to the audio and the text afterwards?

Where the line blurs

The two categories are not as sharp as marketing suggests. Many speech-to-speech products still produce a transcript alongside the audio, and many cascaded products stream partial results between stages so that speech begins before the model has finished writing. Some systems use a native audio model for the conversation and a text step for lookups or rule checks. When a vendor says its product is end-to-end, ask what exactly is end-to-end, and where text still appears.

This matters because the practical questions do not change. Where can you read what was said? Where can a rule be enforced? How is a wrong answer caught? A hybrid that gives you a reliable transcript and a natural voice may be the sensible middle road, provided you can verify both parts.

Testing fairly

Comparisons are easy to rig by accident. Use the same script, the same phone line type and the same background noise for each candidate. Test from a mobile phone in a car or on a street, not only from a quiet room. Include the awkward moments real callers produce: a correction in mid-sentence, a spelled name, a question that is out of scope and a request for a person.

Score each call on a few simple points: was the answer correct, did the agent wait for the caller, how long was the gap before each reply, and how did the call end. A modest set of calls scored consistently tells you more than a polished demo.

How Call approaches this

The practical lesson is that the architecture matters less than the behaviour on your calls. In Call, an AI agent starts speaking as soon as the first part of its answer is ready, prepares greetings in advance on outbound calls and keeps replies short. You set the voice and the transcriber options, test by ringing your own phone and read the conversation afterwards in call history. Whatever the underlying technology, hold it to the same standard: does it answer accurately, hand over politely and sound acceptable to your callers? The measures in How to Evaluate a Voice Agent's Conversations apply to both.

Frequently asked questions

What is a speech-to-speech model?

It is a model that takes spoken audio as input and produces spoken audio as output, without a separate text stage between recognition and speech.

What is a cascaded voice pipeline?

It is a chain of speech recognition, a language model and speech synthesis, with text passed between the stages.

Which is more natural?

Speech-to-speech systems are often described as smoother, but quality differs between products. Judge by listening to calls like yours.

Which is easier to audit?

Cascaded pipelines produce text at each step, so logs, checks and rules are simpler. With end-to-end models, find out what the vendor records and how reliable it is.

Can I mix both approaches?

Some platforms combine them, for example using a cascaded design for certain calls. Ask how a system handles transcripts and rules in whichever mode it uses.

Does the choice affect cost?

It can, but prices and models change, so ask for current terms from the vendor and compare them against measured results rather than headline claims.

Test an AI agent on your own phone line with Call

In the demo console you can try incoming calls in a simulation.

Related guides