What is an AI phone agent?
An AI phone agent is a system that answers (or makes) phone calls using a natural synthesised voice, understands the caller in real time, holds a coherent conversation and carries out actions (booking appointments, logging leads, transferring calls). It works 24/7 and handles several calls at once with no queues.
The typical stack combines three pieces: SIP telephony (Twilio, Zadarma, Vonage), speech synthesis and recognition (ElevenLabs ConvAI, Deepgram, OpenAI Realtime API) and an LLM with access to tools. The delay between the caller finishing speaking and the agent replying needs to be under 700ms for the conversation to feel natural.
Peninsular Spanish voices are now indistinguishable from a human in most cases. The models cope with Andalusian, Galician, Catalan and Latin American accents, callers interrupting, and changes of subject within a single call. What still proves difficult: emotional nuance and very specific cultural context.
An applied example
Example: a vet clinic with dozens of calls a day loses some of them to a busy line or to calls outside opening hours. With an AI phone agent, it picks up on the first ring and works out whether it is an emergency (passed to the on-call human), an appointment request (booked in the calendar, with a confirmation sent by WhatsApp) or a general enquiry (answered from the FAQ). The receptionists can focus on clients in the clinic instead of answering the phone.
When it is worth it
- You get calls every day and a good share of them are repetitive (opening hours, location, prices, appointments).
- You miss calls outside opening hours, at peak times or when the team is busy with something else.
- You have a system (calendar, CRM) the agent can connect to in order to book or look things up.
- Your customers are fine talking to an AI as long as it sorts out their issue or passes them to a human if it can't.
Common mistakes
- Setting the agent never to transfer to a human. It causes frustration when the case is complex.
- Using cheap, low-quality robotic voices. It puts off older customers and damages your brand.
- Not recording calls for continuous improvement. Without logs, you can't tell where the AI is going wrong.
- Paying for the agent during hours when call volume is low. It's better to switch it on only outside opening hours or at peak times, with a low cost per minute.
- Not integrating with WhatsApp to send a post-call summary. The customer hangs up without remembering what was agreed.
How we do it at STAKKER
STAKKER SYSTEMS builds phone agents on ElevenLabs ConvAI + Zadarma (Spain) or Twilio (US). We use a Fastify stack for orchestration, with integration with Google Calendar/Calendly/Cal.com/HubSpot and an automatic post-call summary sent by WhatsApp. Scope and price are agreed after a free diagnostic; we don't work with standard price lists.
Frequently asked questions
Does the voice sound natural in Spanish from Spain?
Yes. ElevenLabs voices in peninsular Spanish are indistinguishable from a human in short conversations. In long, emotional calls you can still notice something, but most people don't realise it's an AI on the first call.
How many calls can it handle at once?
There's no practical limit. If 50 calls come in at the same time, it handles all 50 in parallel. There are no queues.
What happens if the agent doesn't understand the customer?
It detects confusion, frustration or escalation keywords, and transfers the call to the number you choose or alerts you on WhatsApp with the context.
Does it work with my current number?
Yes. With Zadarma (a SIP provider), your current public number stays the same. Internally, the call is routed to the agent.
Related terms
- AI AgentAn AI agent is a software system that combines a large language model (LLM) with access to tools (APIs, databases, messaging) and a goal, so it can carry out tasks with minimal human involvement. The difference from a traditional chatbot is that an agent doesn't just chat: it acts.
- WhatsApp Business APIWhatsApp Business API is Meta's official route for businesses to connect WhatsApp to their systems (CRM, chatbots, AI agents, automation). Unlike the free WhatsApp Business app, the API supports multiple simultaneous users, official templates, automated bots and high message volumes, all in line with Meta's policies.
- AI AutomationAI automation combines traditional workflows (n8n, Zapier, Make) with large language models (LLMs) to carry out tasks that used to need human judgement: sorting messages, drafting replies, summarising documents, deciding escalation routes, creating content. It goes a step beyond the classic "if X happens, do Y".