Voice AI technology

RAG for Voice Agents: Answering from Your Own Content

How retrieval-augmented generation lets a phone AI answer from your documents and FAQs, with tips on preparing content, limiting errors and testing answers.

Call editorial team · Published · 5 min read

A language model knows a lot about the world and nothing about your business. It does not know your opening hours, your cancellation policy or the name of the technician who handles boiler repairs. If you ask it to answer a caller's question without that information, it may guess. Retrieval-augmented generation, usually shortened to RAG, is the standard remedy: find the relevant part of your own content, hand it to the model with the question, and ask it to answer from that material.

A person arranging printed documents and a laptop to prepare a knowledge base for an assistant
Illustrative image

This guide explains how RAG works in the context of phone calls, how to prepare content for it, where it goes wrong and how to test it. For a background on the pipeline, see How Voice AI Works: STT, LLM and TTS.

How RAG works in a call

When a caller asks a question, a voice agent working with RAG typically does this:

The retrieval path

  1. ListenSpeech recognition turns the caller's words into text.
  2. SearchThe system searches your content for passages related to the question.
  3. SelectIt keeps the few most relevant passages and drops the rest.
  4. ComposeThe language model writes a short answer using the question and those passages.
  5. SpeakText-to-speech reads the answer aloud.

Why not give the model everything?

You could try to paste all your content into the instructions. But the model's working space is limited, long instructions slow replies, and a large pile of text makes it easier for the model to miss the one line that matters. Retrieval keeps the working set small and relevant.

What "search" means

Search can be keyword-based, meaning-based (finding passages with similar meaning even if the words differ) or a mixture. Meaning-based search helps with callers who phrase a question in unexpected ways, such as "can I come on Sunday?" when your content says "weekend hours". Whatever method is used, the passages are small chunks of your documents, so the way you write and split content directly affects what is found.

Choosing and preparing content

Quality in, quality out is especially true here. A voice agent can only be as accurate as the material behind it.

Common content sources

  • FAQ and Q&A pairsExact answers to specific questions; the most reliable source for common calls.
  • Web pagesExisting pages about services, hours and policies, read in once or crawled across a site.
  • DocumentsPDFs and word-processor files with price lists, menus or procedures, if the text is selectable.
  • Free textNotes for rules that do not fit anywhere else, such as how to handle a holiday.
  • Corrections from real callsAnswers fixed by a person after listening to a call.

Write for the ear

Content written for reading often sounds odd when spoken. Tables, long lists and abbreviations do not work aloud. Rewrite key answers in plain sentences, with one fact per sentence, and spell out how numbers and times should be spoken. Keep each passage self-contained so that it makes sense without the page around it.

Remove contradictions

If one page says you close at five and another says six, the agent will say one of them at random. Before importing anything, find old pages, drafts and duplicate files, and update or delete them. Add dates to policies so that the newest wins.

Decide what is out of bounds

Tell the agent what it should not answer: legal, medical or financial advice, anything about a specific person's account, or topics you handle only by human. The right response is a polite hand-off. See Guardrails for Voice AI Agents.

Where RAG goes wrong

RAG reduces hallucination but does not eliminate it. Typical failures are:

Failure modes

What happensRemedy
Nothing relevant foundThe agent improvises from general knowledge.Instruct it to say it does not know and offer a person; add the missing content.
Wrong passage foundA similar-sounding policy for a different service is used.Rewrite passages with clear titles; separate similar topics.
Outdated contentThe agent states last year's hours.Give each source an owner and a review date.
Over-long retrievalToo much text slows the reply and confuses the answer.Use shorter, focused passages.
Misheard questionSpeech recognition errors lead to a wrong search.Test with noisy audio and accents; confirm important details.

For a deeper look at preventing invented answers, read Preventing Made-Up Answers in Voice AI. To make the agent look up live facts, such as the status of an order, rather than static text, see Letting a Voice Agent Look Things Up.

The speed budget

Phone conversations tolerate very little delay. Each extra step, such as searching, adds time. Keep sources tidy and small, avoid retrieving more passages than the answer needs, and test response times on real calls rather than on a laptop. Voice AI Latency explains how delays are felt and where they come from.

Keep it current

Hours, prices and policies change faster than anyone remembers to edit the knowledge base. Put a recurring reminder in a shared calendar: before each season, holiday or price change, the content owner reviews the sources and re-indexes them. After a change, call the agent yourself and ask about the thing that changed. A thirty-second check prevents a day of wrong answers.

Test before and after launch

Make a list of questions in three groups: questions the content answers clearly, questions it answers partly, and questions it does not cover. For each, record what the agent says. Check that it is correct, short enough to say aloud, and that for the third group it admits it does not know. Re-run the list whenever the content changes.

A test routine

  • Ask each common question in two or three different wordings.
  • Ask the same questions with a noisy line or a strong accent.
  • Ask for something the content does not contain.
  • Ask a question with a trap, such as an old price or a closed day.
  • Listen for answers that are too long to follow by ear.
  • Review real call transcripts weekly in the first month.

Corrections that stick

When a human finds a wrong answer, fixing the transcript is not enough; the correction should go back into the knowledge base so the agent answers correctly next time. Decide who is allowed to make that change and keep a history of changes.

Doing it in Call

In Call, knowledge belongs to an AI agent and is added under Sources:

Storage is shared across the organisation, with limits per source, and only workspace managers can add, change or delete sources. A banner shows when live calls do not yet use the knowledge base. Call history lets you correct an answer the agent gave and, if you choose, save it as the correct answer for next time. Start with Using Knowledge Sources in Call and Building a Knowledge Base for Support and AI.

Frequently asked questions

Is RAG the same as training a model on my data?

No. Training changes the model itself. RAG leaves the model unchanged and gives it relevant passages from your content at the moment of the question, which is easier to update and easier to check.

Does RAG stop a voice agent from making things up?

It reduces the problem but does not remove it. The agent can still misread a passage or answer a question your content does not cover. Clear instructions, good content and regular testing keep errors low.

How much content do I need?

Less than most people expect. A few dozen accurate Q&A pairs that cover your most common calls often outperform a large unedited archive. Add more as real calls show gaps.

Can the agent use content in PDFs?

Usually, if the PDF has selectable text. Scanned image-only PDFs need text recognition first. Check what the system extracted, because tables and columns may come out jumbled.

How often should I update the knowledge base?

Whenever facts change, and on a regular review date for everything else. Give each source an owner so updates are not left to chance.

Add knowledge sources to your AI agent and check its answers in Call

In the demo console you can try incoming calls in a simulation.

Related guides