A voice AI agent answers a phone call by connecting the call to a realtime speech model through a media server. The model listens, decides what to say, and calls small, well-defined functions in your own system, such as "find a free slot" or "book this time". Everything that touches your data goes through those functions, not through the model.
This is the architecture we use for appointment setters. Here is what each part does and why it is there.
How does the call reach the AI?
A normal phone number is routed to a SIP trunk, for example from Twilio. SIP is the standard that carries phone calls over the internet. The trunk forwards the call into a media server such as LiveKit, which puts the caller and the AI agent in the same audio room.
The caller does not install anything. To them it is an ordinary phone call. The same agent can also take calls from a "talk" button on a website, because the browser joins the same kind of room.
What does the speech model do?
Older voice bots chained four separate services: one to detect when someone is speaking, one to turn speech into text, a language model to decide the reply, and a voice engine to read it out. Each hop added delay, and the caller heard the gaps.
Realtime speech models, such as Gemini Live, handle all four inside one model. That cuts the pauses and makes interruptions feel natural. If the caller starts talking while the agent is speaking, the agent stops and listens, the way a person would.
How does the agent book a real appointment?
The model never writes to the calendar directly. It is given a short list of tools, and each tool is a function in the backend, usually a FastAPI service:
- Check availability: returns open slots for a date and a person.
- Hold a slot: reserves a time for a few minutes while the caller decides.
- Book: turns the hold into a confirmed appointment.
- Save the caller: stores the name, number and reason if the call ends early.
Because the backend owns these actions, it can enforce the rules the model might forget. Two callers cannot take the same slot, bookings outside opening hours are refused, and every action is logged.
Why hold a slot instead of booking straight away?
Two people can ask for the same time at the same moment. If the system only reads the calendar, both see the slot as free and both get booked. A hold reserves the time while one caller confirms, then either becomes a booking or is released. We wrote more about this in how an AI appointment setter avoids double-booking.
How do you stop the agent saying the wrong thing?
Three layers do most of the work:
- A narrow script. The instructions describe one job: find out what the caller needs and book a time. The agent is told what it must not do, such as quoting prices it has not been given.
- Tools as the only source of facts. Opening hours, names and free slots come from tool results, not from the model's memory.
- A clear handoff. When a request is urgent or outside the script, the agent says so and passes the caller to a person or takes a message.
What does it take to run this in production?
The parts that are easy to skip in a demo are the ones that matter on a real phone line:
- Tenant isolation, so one business never sees another business's calls or settings.
- Encrypted credentials for telephony accounts, decrypted only when a call needs them.
- Retries and timeouts around every external service, so one slow response does not drop a call.
- Health checks that let the hosting platform restart a stuck worker before callers notice.
- Confirmation emails to both sides, so a booking made by voice also exists in writing.
Is a voice agent right for every business?
No. It works best where calls follow a pattern, such as booking, rescheduling and simple questions, and where missed calls cost real money. If most calls are complex conversations, a better first step is a short call-back flow: the agent takes the details, and a person rings back.
The Bookhatch AI case study shows this pattern in a healthcare appointment-setting product.
Delivery and AI engineering