Voice AI technology

How Voice AI Works: Speech Recognition, LLM and Text-to-Speech

A plain-English tour of the pipeline behind voice AI phone agents: speech-to-text, a language model, text-to-speech, plus turn-taking, delay and safe handoffs.

Call editorial team · Published · 5 min read

A phone call with an AI agent can feel like magic or like a puzzle, depending on whether it works. Behind it sits a fairly simple idea: turn the caller's voice into text, let a language model decide what to say, and turn the reply back into speech, all fast enough to feel like a conversation. This guide explains each stage, where things go wrong and what that means for how you set up an agent. For the wider picture, start with What is AI phone reception?.

A phone beside a diagram of sound waves turning into text and back into speech
Illustrative image

The basic loop of a voice AI call

  1. ListenAudio from the caller is captured and cleaned.
  2. TranscribeSpeech-to-text turns the audio into words.
  3. DecideA language model chooses what to say or do next.
  4. SpeakText-to-speech turns the reply into audio.
  5. RepeatThe loop runs for every turn until the call ends or is handed over.

Stage one: speech-to-text

Speech recognition converts sound into words. On a phone line the audio is narrow-band and compressed, which makes it harder than studio recording. Accents, background noise, speaking over each other and unusual names all affect accuracy. Modern systems can do well on everyday speech, but numbers, spellings, names and domain words such as medicines or street names are common weak points.

More detail is in speech recognition for phone calls, handling numbers and names and accents and dialects.

Stage two: the language model

A large language model reads the transcript so far, along with instructions you wrote and any information you supplied, then produces the next reply. It does not look things up by itself; it relies on what it was given. That is why the prompt and the knowledge base matter so much. The model can sound fluent while being wrong, which is why guardrails and testing are essential.

What shapes the model's reply

  • InstructionsThe role, tone, rules and limits you give it.
  • KnowledgeDocuments, Q&A and facts it may draw on.
  • Conversation so farWhat has been said in this call.
  • ToolsActions it may take, such as looking up an order or noting a message.

Guides such as prompting a phone agent, RAG for voice agents, hallucinations and guardrails go deeper. A prompt that says what to do when unsure, for example offering a person or taking a message, prevents many problems.

Stage three: text-to-speech

Text-to-speech turns the reply into audio. Voices vary in clarity, warmth and how they handle pauses, numbers and names in your language. Some can be tuned or adjusted with a pronunciation dictionary. Pick a voice for clarity on a phone line rather than for novelty, and test it with the actual phrases it will say. See text-to-speech in Japanese for an example of language-specific issues, and voice cloning for consent and risk when using a custom voice.

Why timing decides the experience

Human conversation has fast turns. A gap of more than a second or so feels odd. In a three-stage pipeline each step adds delay, and the model may take time to think. Designers reduce delay by streaming: starting to speak before the whole reply is ready, and starting to transcribe before the caller stops. They also tune when to decide that the caller has finished.

Pipeline vs speech-to-speech

Three-stage pipelineSpeech-to-speech model
How it worksSeparate recognition, model and voiceOne model handles audio in and out
ControlEach stage can be tuned or swappedLess visible, more integrated
DelayAdds up across stagesCan be lower
TranscriptText available at every stepMay need extra work to produce text
MaturityWidely usedNewer, changing quickly

Speech-to-speech models are a newer approach. Either way, latency, interruptions and turn-taking determine whether the agent feels natural.

Interruptions

People interrupt. A good agent stops talking when the caller starts, and does not take a cough or a background voice for an interruption. This depends on detecting speech, covered in voice activity detection. Poorly tuned systems either talk over callers or wait too long.

Tools, knowledge and actions

A voice agent becomes useful when it can do more than chat. With connected tools it can look up information, record a message, note a callback request or pass the call to a person. Each tool should have a narrow purpose and clear rules. An agent can take a request for an appointment; confirming it is a different matter, and many businesses prefer a person to confirm. Function calling explains how tools are described to the model.

What happens to the audio and the text

Each stage handles personal data. The audio holds a voice; the transcript holds what was said, often including names, addresses and reasons for calling. Before you deploy, find out where each stage runs, what is stored, for how long and who can read it. Decide whether sensitive details, such as payment or health information, should be collected by the agent at all or passed to a person. Designing voice AI with privacy in mind covers this in more depth.

Keep the agent's job narrow. An agent that answers common questions, takes a message and hands over is easier to test, easier to explain to callers and easier to keep within your privacy promises than one that tries to do everything.

What to expect in practice

Expect a good result on simple, predictable calls: opening hours, directions, basic questions, taking a message. Expect more difficulty with long, emotional or ambiguous calls, heavy accents, bad lines and requests that need judgement. A sensible rollout starts with a small set of call types, listens to every call at first, then widens as confidence grows. The phrase to remember is that the agent is designed to help with defined tasks, not to replace judgement.

Where it goes wrong

Design for these. Make it easy to reach a person, keep answers short, confirm important details such as numbers and dates, and review real calls regularly. Testing a voice agent and evaluation metrics explain how.

Doing it in Call

In Call, an AI agent is built from a persona and prompt, a voice, a language, and knowledge sources, then tested on screen and by phone before it is switched on for a line. Call lets you choose among voices, tune pronunciation with a dictionary, adjust how the speech is recognised and keep sources up to date. The agent is designed to put callers through to a person when asked, and it can take a message. Until you switch an agent on for a line, calls ring your team. Always test with your own calls first. Setting Up an AI Agent in Call walks through the steps.

Frequently asked questions

What are the three parts of a voice AI agent?

Speech-to-text to hear the caller, a language model to decide what to say and text-to-speech to speak the reply. Some newer systems combine them.

Why does the AI sometimes pause before answering?

Each stage takes time, and the system must decide the caller has finished speaking. Streaming and tuning can shorten the gap but not remove it.

Can the AI make things up?

Yes. A language model can produce plausible but wrong answers. Give it good knowledge, clear limits and an instruction to offer a person when unsure.

Can a voice agent book appointments?

It can take a request and pass it on. Many businesses prefer a person to confirm, so decide what the agent is allowed to promise.

Do callers need to be told it is an AI?

In many places people expect or are entitled to know. Say so clearly at the start of the call and check the rules that apply to you.

How do I check that it works?

Test with real calls in different accents and noise levels, listen to the results and keep a list of questions you rerun after every change.

Try an AI voice agent on your own line in Call

In the demo console you can try incoming calls in a simulation.

Related guides