Anatomy of a Healthcare Voice Agent: From Phone Ring to EHR Write

When people hear “AI receptionist” they picture a model talking. The model is the smallest part of the system. Most of the engineering is in everything around it: getting audio in and out reliably, giving the agent tools it can’t misuse, writing correct data into a clinic’s medical record system, and knowing afterwards what happened on every call.

This is a tour of how a call flows through the platform I work on at Simbie AI, at the level of architecture rather than any specific clinic or vendor contract.

Layer 1: telephony and the voice runtime

Twilio owns the phone number and the PSTN side: receiving the call, SIP, and executing transfers. Retell AI owns the real-time voice loop: streaming speech-to-text, deciding when the caller has finished speaking, calling the LLM, handling interruptions, and text-to-speech.

We deliberately don’t build this layer. Low-latency turn-taking is hard and is not where our product is different. What we do own is how it’s configured: the prompt, which functions the agent can call, and what happens when it calls them.

Layer 2: the agent backend

When the agent needs to do something (look up a patient, find a slot, transfer the call) the voice runtime calls one of our custom functions over HTTPS. Everything interesting happens behind that call.

The function gateway

Every function call enters through one gateway that does the same four things:

1. Authenticate the request as coming from the voice runtime for a specific agent and call.
2. Validate arguments against a schema. LLMs produce malformed arguments more often than you’d like: a date in the wrong format, a missing field, a state written out in full where the EHR wants a two-letter code.
3. Normalise what can be normalised safely (that state name becomes OH) before anything reaches the EHR.
4. Return errors the model can act on. “Invalid arguments” teaches the model nothing. “date_of_birth must be YYYY-MM-DD; you sent ‘March 3rd 1980’” usually gets a correct retry.

export const bookAppointment = defineFunction({
  name: "book_appointment",
  input: z.object({
    patientId: z.string(),
    providerId: z.string(),
    slotStart: z.string().datetime(),
    reason: z.string().max(200),
  }),
  async handler(args, ctx) {
    const slot = await ctx.ehr.getSlot(args.providerId, args.slotStart);
    if (!slot?.available) {
      return { ok: false, message: "That slot was just taken. Offer the next two available times." };
    }
    const appt = await ctx.ehr.book({ ...args, source: "voice-agent", callId: ctx.callId });
    await ctx.callState.record("appointment_booked", { appointmentId: appt.id });
    return { ok: true, message: `Booked for ${formatForSpeech(appt.start)}.` };
  },
});

Note the return values are written for the model to say, not for a developer to read. That one habit removes a lot of awkward agent behaviour.

The agent harness

The harness decides which tools and instructions are active at each point in the call. A caller booking a new-patient appointment needs different tools than one asking for a prescription refill. Giving the model every tool all the time makes it worse at choosing; the harness scopes tools per scenario and pins the exact agent configuration version used for each call, so we can always answer “what was the agent running when this happened?”

The EHR adapter

This is where healthcare is different from other integrations. The EHR is the system of record for patient care. Writes must be correct, attributable and recoverable. The adapter:

• Wraps each EHR’s API (FHIR where available, vendor APIs where not) behind one interface: patient search, new-patient registration, provider resolution, slot lookup, booking, and filing documents to the chart.
• Matches patients conservatively. Name plus date of birth plus one more identifier; if it’s ambiguous, the agent asks, or the call is handed to a human.
• Tags every write with the call id, so anything the agent did can be traced back to a transcript.

Call state

Each call has a state record (Postgres for durability, Redis for the hot path within a call) that tracks what the agent has done: patient matched, appointment booked, transfer attempted. Functions are idempotent against it: if the voice runtime retries book_appointment, the second call sees the booking already exists and returns the same result.

Layer 3: after the hang-up

The call ending is the start of the second half of the system. The voice runtime posts webhooks (call_ended, then call_analyzed) and a pipeline takes over:

• Rule-based call properties and scenario codes: was a patient matched, was an appointment booked, was the call transferred, and why.
• A deterministic review step that checks outcomes against what the EHR now says, rather than trusting the model’s own summary.
• LLM summaries and data extraction for staff, on scrubbed transcripts.
• Analytics that compare agent versions on the same scenarios.

Observability runs across all three layers with OpenTelemetry traces keyed by call id, so a single failed booking can be followed from the first ring to the EHR response.

Where it breaks

If I had to rank where production issues come from, it would be:

1. The EHR edge. Validation rules nobody documented, slow endpoints, and data that’s technically valid but wrong.
2. Webhook ordering. Events arriving out of order or not at all. (That one deserves its own post.)
3. Tool design. Tools that are too broad, or return messages the model can’t use.
4. The model. Usually last, and usually fixed by improving one of the three above.

That ranking is the main thing I’d want someone new to voice agents to take away. The agent is only as good as the system it’s allowed to act on.