Voice AI technology

Prompt Design for Phone AI Agents: Instructions That Hold Up

How to write instructions for a voice AI agent that answers phone calls: role, tone, allowed answers, handoff rules, spoken style and how to test them.

Call editorial team · Published · 5 min read

The instructions you give a voice AI agent decide most of how it behaves on a call. Vague instructions produce vague, sometimes wrong, conversations. Over-long instructions bury what matters. Good instructions are short, specific and tested.

A person reviewing written instructions on a laptop with a headset on the desk
Illustrative image

This guide is for people writing those instructions, often called a system prompt, for an agent that answers a business phone line. It assumes you know the basics from What Is AI Phone Reception? and How Voice AI Works. The aim is not clever wording but a reliable, honest agent.

Why phone prompts differ from chat prompts

A chat reply can be long, formatted and full of links. A spoken reply cannot. Callers hear each sentence once, in sequence, while waiting for the next. Everything about a phone prompt follows from that.

Chat versus phone

Text chatPhone call
LengthParagraphs are fineOne or two short sentences at a time
FormatLists, links, boldNone of these; speak plainly
ErrorsReader can scroll backCaller cannot; misheard details are costly
InterruptionsRareCommon; callers talk over the agent
Names and numbersTypedSpoken and heard, so easily wrong

A structure that works

Keep instructions in clear parts. Order them so the most important things come first.

Parts of a good phone-agent prompt

  1. RoleWho the agent is and for which business.
  2. GoalWhat a good call looks like.
  3. ToneHow it should sound, such as calm and friendly.
  4. Facts and sourcesWhat it may rely on and what to do when it does not know.
  5. RulesWhat it must always or never do.
  6. HandoffWhen and how to pass the call to a person or take a message.

Role and goal

One or two sentences are enough: "You answer the phone for a dental clinic. Your goal is to help callers with opening hours and directions, and to take a request for an appointment or a message for the team." Avoid grand personalities. A clear job beats a clever character.

Tone

Name the tone you want and keep it consistent: calm, polite, concise. If your business has a style, such as formal Japanese honorifics, say so explicitly. Do not ask for humour or chattiness unless it suits your callers.

Writing for spoken replies

Facts, limits and "I don't know"

The biggest risk with a language model is a confident wrong answer. Reduce it by giving the agent trusted material and a clear rule for gaps.

Four rules that prevent most problems

  • Answer only from supplied materialHours, services and policies that you provide and keep up to date.
  • Say when it does not know"I cannot confirm that on this call; I can take a message or put you through."
  • Never invent specificsNo prices, dates or promises that are not in your material.
  • Stay in scopePolitely decline topics outside the business.

For larger sets of information, see Retrieval for Voice Agents and Voice AI Guardrails.

Be careful with bookings and commitments

An AI agent should not confirm a booking, give a discount or promise a callback time on its own unless your system truly supports it and you have decided it should. A safer pattern is that the agent takes the request, reads it back, and says that the team will confirm. State this in the prompt in plain terms.

Handoff and escalation

Say exactly when the agent should hand over. Typical triggers are: the caller asks for a person, the caller is upset, the matter is urgent or sensitive, or the agent has failed to help after a couple of attempts. Also say what to do when nobody is available: take a message, offer a callback, or give opening hours. Connect it with Letting Callers Reach a Person Quickly.

Handoff rules to include

  • The caller asks for a person
  • The caller sounds upset or reports an emergency
  • The question is outside the agent's material
  • The agent has misunderstood twice in a row
  • The topic involves payment, legal or medical decisions
  • What to do if no one is available

A short example prompt

Here is a deliberately plain example for a hypothetical clinic. Adapt the wording; do not copy it unchanged.

Notice what is absent: no long backstory, no promises about speed, and no instruction to be impressive. Each line either defines the job, limits it or says how to hand over. As you learn from calls, add rules for the real problems you see, and remove lines that no longer earn their place.

Testing and revising

A prompt is a draft until it has met real conversations. Test deliberately and change little at a time.

  1. Place practice calls covering typical questions, odd questions, silence and interruptions.
  2. Include callers with accents, noise and unusual names.
  3. Read the transcripts and note exactly where the agent went wrong.
  4. Change one thing in the prompt, then run the same calls again.
  5. Save a copy of each version that works, so you can roll back.

Finally, treat the prompt as living documentation. Date each version, note why a line was added, and review the whole prompt now and then to remove rules that no longer matter.

Doing it in Call

In Call, an agent's instructions sit in its system prompt, read together with its persona and the knowledge it can use. Persona settings include a name, role, tone, language, goals and rules, and a choice of model that trades speed against capability. The agent is built to answer only from your instructions, its attached script and matching passages of its knowledge, to say plainly when it cannot confirm something and to offer a person where the line can put callers through. It replies in short spoken sentences. Instructions cannot switch off these built-in rules. Test calls and the playground let you hear the result, and restore points save a copy of an agent before a big change so you can roll back. See Setting Up an AI Agent in Call.

Frequently asked questions

How long should a phone agent prompt be?

As short as it can be while covering role, goal, tone, sources, rules and handoff. Long prompts bury the important instructions.

Why should replies be so short?

Callers hear each sentence once and cannot scroll back. Short replies are easier to follow and let the caller interrupt naturally.

How do I stop the agent making things up?

Supply approved material, tell it to answer only from it, and give it a standard phrase for when it cannot confirm something.

Can the agent confirm appointments?

Only if your system and rules truly support it. A safer default is that it takes the request and the team confirms.

How should I test a prompt?

Place practice calls with ordinary and awkward questions, read the transcripts, change one thing and repeat.

What should I do if a change makes things worse?

Roll back to the last working version. Keeping a saved copy of each good version makes this quick.

Write and test instructions for your AI phone agent in Call

In the demo console you can try incoming calls in a simulation.

Related guides