Speech-to-Speech Models vs Cascaded Voice Pipelines Compared
Two ways to build a phone voice agent: speech-to-speech models and cascaded pipelines. What each does well, where each is weak and how to choose by testing.
When you hear a voice agent speak, one of two designs is usually behind it. In the first, software turns the caller's speech into text, a language model writes a text reply, and a voice engine reads it aloud. In the second, a single model listens to audio and answers in audio. Vendors argue about which is better. The honest answer is that each trades something.

This guide compares the two in plain language so you can ask better questions when choosing a platform. For a wider introduction, start with How Voice AI Works on a Phone Call.
The cascaded pipeline
A cascaded pipeline, sometimes called speech-to-text, then model, then text-to-speech, has been the standard approach for years. Each stage is a separate component, and text passes between them.
Cascaded pipeline
- Speech recognitionConverts the caller's audio into text.
- Language modelReads the text, with instructions and sources, and writes a reply.
- Speech synthesisTurns the reply text into spoken audio.
- TelephonyCarries the audio to and from the caller.
Because the middle of the system is plain text, you can read it, search it, store it, check it against rules and replace any stage. You can swap in a recogniser that is better with your callers' accents or a voice that suits your brand.
The speech-to-speech approach
A speech-to-speech model, also described as an end-to-end or native audio model, handles audio directly. It listens to the sound itself and generates sound as output, though many such systems also produce text for logging.
Speech-to-speech
- Audio inThe caller's voice goes straight to the model.
- One modelIt interprets meaning, tone and timing and plans a reply.
- Audio outIt generates the spoken response directly.
- TelephonyThe audio travels to and from the caller.
The appeal is that nothing is lost between stages. A pause, a laugh, hesitation or a sigh can influence the reply, and the model can start speaking sooner because it is not waiting for three separate steps.
Side by side
Cascaded versus speech-to-speech
| Cascaded | Speech-to-speech | |
|---|---|---|
| Naturalness | Depends on each stage; can feel stepwise | Often smoother, with timing and tone preserved |
| Response time | Sum of three stages, which can be tuned | Potentially shorter, since fewer hand-offs |
| Control | High: rules can be applied to text at each step | Lower: harder to inspect what happens inside |
| Transcripts and logs | Natural by-product | Usually generated separately and may differ from what was said |
| Swapping parts | Easy: change recogniser or voice alone | Limited to what the model offers |
| Voice choice | Wide choice of voices | Often a smaller set |
| Maturity | Long track record | Newer, changing quickly |
Treat any claim of absolute superiority with caution. Quality varies between products inside each category, and results on a clean demo may not hold on a noisy mobile call.
What matters for phone calls
Phone audio is narrow and noisy
Telephone audio is compressed and often noisy. Whichever design you choose, check how it performs on real phone audio and not just on studio recordings. Why Phone Audio Is Hard for Speech Recognition explains the effect.
Control and auditability
If you must demonstrate what an agent said and why, text at each stage helps. A cascaded design lets you apply a check between the model's reply and the voice. With an end-to-end model you rely more on the vendor's controls and on testing. Review Guardrails for Voice AI Agents and decide what you need to enforce.
Interruptions and timing
Native audio models often handle overlap and interruptions more fluidly because they hear the caller continuously. Cascaded systems rely on detection of speech and silence to decide when to stop and start; see Voice Activity Detection and Turn-Taking in Voice Agents. Better engineering can narrow the gap, and the gap may keep changing.
Languages and accents
Check the languages and accents that matter to you by testing, rather than accepting a list in a brochure. A model strong in one language may be weaker in another, and the same holds for individual stages in a pipeline.
Choosing for your situation
A few hypothetical scenarios show how priorities pull in different directions.
- A small clinic that needs every call logged with a reliable transcript and exact wording on a few topics may prefer a cascaded design for its control.
- A concierge line where a warm, fluid conversation is the whole point may value the naturalness of a speech-to-speech model, if testing shows it holds its facts.
- A business that wants to change voices or languages often may prefer the flexibility of separate components.
- A team with strict internal review needs may need to confirm how any end-to-end system records and exposes what was said.
Questions to ask a vendor
- Which stages does the system use, and can I change them?
- Do I get a transcript of every call, and is it what the model heard?
- How are rules and limits enforced?
- How does it behave on mobile and noisy lines?
- What are typical response times on real phone calls?
- How are interruptions handled?
- What happens to the audio and the text afterwards?
Where the line blurs
The two categories are not as sharp as marketing suggests. Many speech-to-speech products still produce a transcript alongside the audio, and many cascaded products stream partial results between stages so that speech begins before the model has finished writing. Some systems use a native audio model for the conversation and a text step for lookups or rule checks. When a vendor says its product is end-to-end, ask what exactly is end-to-end, and where text still appears.
This matters because the practical questions do not change. Where can you read what was said? Where can a rule be enforced? How is a wrong answer caught? A hybrid that gives you a reliable transcript and a natural voice may be the sensible middle road, provided you can verify both parts.
Testing fairly
Comparisons are easy to rig by accident. Use the same script, the same phone line type and the same background noise for each candidate. Test from a mobile phone in a car or on a street, not only from a quiet room. Include the awkward moments real callers produce: a correction in mid-sentence, a spelled name, a question that is out of scope and a request for a person.
Score each call on a few simple points: was the answer correct, did the agent wait for the caller, how long was the gap before each reply, and how did the call end. A modest set of calls scored consistently tells you more than a polished demo.
How Call approaches this
The practical lesson is that the architecture matters less than the behaviour on your calls. In Call, an AI agent starts speaking as soon as the first part of its answer is ready, prepares greetings in advance on outbound calls and keeps replies short. You set the voice and the transcriber options, test by ringing your own phone and read the conversation afterwards in call history. Whatever the underlying technology, hold it to the same standard: does it answer accurately, hand over politely and sound acceptable to your callers? The measures in How to Evaluate a Voice Agent's Conversations apply to both.
Frequently asked questions
What is a speech-to-speech model?
It is a model that takes spoken audio as input and produces spoken audio as output, without a separate text stage between recognition and speech.
What is a cascaded voice pipeline?
It is a chain of speech recognition, a language model and speech synthesis, with text passed between the stages.
Which is more natural?
Speech-to-speech systems are often described as smoother, but quality differs between products. Judge by listening to calls like yours.
Which is easier to audit?
Cascaded pipelines produce text at each step, so logs, checks and rules are simpler. With end-to-end models, find out what the vendor records and how reliable it is.
Can I mix both approaches?
Some platforms combine them, for example using a cascaded design for certain calls. Ask how a system handles transcripts and rules in whichever mode it uses.
Does the choice affect cost?
It can, but prices and models change, so ask for current terms from the vendor and compare them against measured results rather than headline claims.