What building 30 AI voice agents taught me.

Almost nothing that decides whether an AI voice agent works has anything to do with the AI. It is the escalation path, the logging, the confirmation message, and the decision about what the agent is not allowed to do. The model is the easy part and it is the part everyone talks about.

That is the short version. What follows is the longer one, including the parts that did not work, because those are where the useful information is.

The first one was held together with Zapier

I did not arrive at this from machine learning. I came from building operational software for businesses that needed something to work on Monday.

The first agent was a hacky phone integration bolted to a Zapier flow. It functioned, roughly. Some callers hung up. It taught me more than the next five did, because everything it lacked turned out to be the thing that matters: it had nowhere to send a call it could not handle.

Why is the AI the easy part?

Because a language model is a component, and a phone call is a process. The questions that decide whether the thing works are not about the model at all.

An agent with a capable model and no escalation path is worse than voicemail, because voicemail at least fails obviously.

What do business owners actually want?

Not AI. This surprised me more than it should have.

I once spent ten minutes explaining language models to the owner of a restaurant group. He waited, politely, and then asked whether it would stop the phone ringing while his staff were serving. That was the whole requirement. Everything I had been saying was irrelevant to it.

Now I start at the outcome. Fewer missed calls. Appointments that get booked at nine in the evening. Staff who are not interrupted mid-service. If the mechanism happens to be AI, that is an implementation detail, and it is the part the buyer cares about least.

Why do the simplest agents outlast the clever ones?

Because every branch is somewhere a caller can get lost. The agents I built early were the sophisticated ones — multi-step logic trees, intent switching, layered fallbacks. They broke more than they helped.

The ones still running years later are almost dull.

The clever version	The one still running
Handles any topic a caller raises	One job, understood properly
Branching logic for every path	Two or three outcomes, clearly offered
Recovers from confusion by re-prompting	Escalates to a person
Impressive in a demonstration	Unremarkable on a Tuesday

Complexity can be added later, once you have watched a hundred real calls and know which paths people actually take. Adding it first means designing for callers you have imagined rather than callers you have heard.

What one of them actually does

A salon, two stylists, no receptionist. Before, a call arriving mid-treatment meant somebody put the scissors down.

The agent answers first now. It books appointments, moves them, cancels them, and answers questions about prices and what the salon offers — completing them during the call rather than taking a message for someone to action later.

It handles around 85% of calls without escalating to a person. That is less impressive than it sounds: booking, changing, cancelling and asking what something costs is most of why anyone rings a salon in the first place. Cover those four and the majority of the volume goes with them. The interesting number was never the 85% — it is what happens to the other 15%.

The part worth describing is the escalation logic, because none of it is AI.

• It reads the live calendar before deciding what to do with a caller who needs a person
• If the diary shows them mid-appointment, it offers a callback rather than interrupting
• If there is a gap right now, it puts the call straight through, exactly as before
• If the caller asks for a person, it escalates regardless of what the diary says

That is four rules and a calendar lookup. There is no cleverness in it at all, and it is the reason the thing is still running.

The unglamorous half: it is integrated with Cal.com for Teams, which meant moving the salon off a paper diary as part of the same job. An agent cannot book into a paper calendar. So the real project was as much about the diary as it was about the voice — which is true of most of these, and is never what the conversation starts out being about.

The rule I will not bend: never let it deny it is AI

Every agent I build says it is an AI assistant, and none of them will deny it if asked. Not once, not as a joke, not to keep a call flowing.

The commercial argument for pretending is that callers respond better to a human. In practice most people work it out within two or three exchanges, and the ones who ask directly are testing precisely that. Getting caught in the lie does not damage trust in the technology. It damages trust in the business whose name is on the phone line — which is my client, not me.

There is now a regulatory argument as well. The EU AI Act’s Article 50 transparency obligations took effect on 2 August 2026 and require that people are informed they are interacting with an AI system, in a clear and distinguishable way, at the latest at the first interaction. Automated phone systems are explicitly in scope.

What I did not expect is that disclosure makes the calls better. People who know they are talking to a system speak more plainly, give information in a more usable order, and are noticeably less annoyed when it asks them to repeat something.

What happens when it gets something wrong?

It mishears names, misreads dates, and occasionally answers a question nobody asked. Any account of this work that leaves that out is selling something.

The thing that separates a system I am comfortable putting in front of a client’s customers from one I am not is whether mistakes are visible and correctable. That means every call logged and transcribed, a person able to take over, and a named human who is accountable for putting it right.

It is the same standard as a new member of staff. You do not expect a faultless first month. You expect to find out quickly when something goes wrong, and to be able to fix it.

What actually moved the needle

Consistently the least interesting parts. Not the model, not the voice quality, not the clever handling of an edge case.

• An SMS or email confirmation sent immediately after the call, so the caller has something in writing
• Matching the agent’s tone to how the business actually speaks, rather than to a generic professional register
• Listening to real calls weekly for the first month and adjusting what it says
• Getting the booking into the diary the business already uses, instead of a report somebody then re-keys
• Being specific about opening hours, areas covered and what the business does not do

None of that is exciting and all of it is the difference between a system that gets used and one that gets switched off after a fortnight.

What I would do differently

Three things, in order of how much they cost me to learn.

Define the failure path before writing a single prompt. It is the part that gets left until last and it is the part that decides whether the thing is safe to deploy.

Scope one job, not a role. "Handle our phones" is not a specification. "Answer out-of-hours enquiries, establish whether it is urgent, book a callback or escalate" is.

Watch real calls earlier. Every assumption I have made about how people phrase things has been wrong at least once, and I would not have found out from testing it myself.

If you are weighing this up for your own business, the companion question is which workflow to start with at all — what to automate and what to keep human covers that, and what to automate first covers the ordering.

Common questions

What is the hardest part of building an AI voice agent?

Not the AI. The hardest part is everything around it: where the call goes when the agent cannot help, who gets notified, what is written back into the business's systems, and what the agent is not permitted to do. A capable model with no escalation path is worse than a voicemail, because voicemail at least fails obviously.

Should an AI agent tell callers it is AI?

Yes, always, and it should never deny it when asked. Callers usually work it out anyway, and being caught pretending damages trust in the business rather than in the technology. Since 2 August 2026 the EU AI Act's Article 50 transparency obligations also require that people are informed they are interacting with an AI system, at the latest at the first interaction.

Why do simpler agents outperform complex ones?

Because every branch is a place to get lost. Multi-step logic trees, intent switching and layered fallbacks all increase the number of states a live caller can end up in, and callers do not follow the paths you designed. An agent with one job, a short set of outcomes and a fast route to a person handles reality better than one built to handle everything.

What happens when the agent gets something wrong?

It will. It mishears names, misreads dates and occasionally answers a question nobody asked. What matters is whether that is visible and correctable: every call logged and transcribed, a person able to take over, and someone accountable for putting it right. Perfection is not the standard for a new member of staff either.

How long does it take to build one that works?

A single, well-scoped agent handling one job is usually weeks rather than months. What extends it is not the model, it is the decisions around it: what qualifies as an emergency, which system owns the booking, what the agent should say when it does not know. Those are business decisions and only the business can make them.

How does an AI receptionist know when to put a call through to a person?

It should be a rule, not a judgement. On one salon build the agent reads the live calendar before deciding: if the diary shows the team mid-appointment it offers a callback, if there is a gap right now it transfers the call straight through, and if the caller asks for a person it escalates regardless. That agent completes around 85% of calls itself, so the escalation rules only govern the remaining 15% — but those are the calls where getting it wrong is most visible.