Running 187 Voice Agents From Config: A Multi-Agent Harness With Versioning

The first voice agents on our platform were each a long prompt and a list of functions, configured directly in the voice runtime. That works for a few agents. By the time we had well over a hundred, across clinics with different specialties, offices, hours and EHRs, it had stopped working in very specific ways:

• Copy-paste drift. A fix to how agents handle insurance questions had to be applied to dozens of prompts by hand, and some always got missed.
• No history. When a clinic said “it was better last week”, there was no reliable way to see what changed.
• One giant prompt. Every agent carried instructions for every situation, which made it slower and worse at each one.

The platform now runs 187+ production voice agents from configuration. This post describes the harness that makes that manageable.

An agent is a document, not a prompt

Each clinic’s agent is defined in YAML: which sub-agents it has, what each one is for, which tools each may use, which model it runs on, and the clinic-specific facts it needs.

agent: example-clinic-main
extends: base/outpatient-v3          # shared behaviour, maintained once
clinic:
  timezone: America/New_York
  offices: [north, downtown]
  ehr: { adapter: fhir-r4, tenant: example }
 
agents:
  intake:
    goal: Identify the caller and the reason for the call
    tools: [search_patient, register_new_patient]
    model: { provider: openai, name: default-fast }
    hands_off_to: [scheduling, transfer]
  scheduling:
    goal: Find and book, reschedule or cancel appointments
    tools: [find_slots, book_appointment, cancel_appointment]
    model: { provider: bedrock, name: default-reasoning }
    hands_off_to: [transfer]
  transfer:
    goal: Route the caller to the right person with context
    tools: [transfer_call, take_message]

Three things about this shape matter:

Inheritance. Most behaviour lives in shared bases like base/outpatient-v3. A clinic config overrides only what’s actually different about that clinic. The insurance-question fix is made once, in the base, and reaches every agent that extends it on their next version.

Schema validation. Configs are validated before they can be saved: unknown tools, a hand-off to an agent that doesn’t exist, or a missing EHR adapter fail immediately, not on a live call.

Small agents. Each sub-agent has one goal and a short allow-list of tools. The orchestrator switches between them as the call progresses. Smaller instruction sets and fewer tools made each step faster and more reliable than one agent that knows everything.

Versions are immutable

Saving a config creates a new, immutable version in the registry. Publishing pins a phone number to a specific version. Nothing changes in production until someone publishes, and rolling back is pinning the previous version again.

Every call records the version it ran on. That single field is what makes the rest possible: “it was better last week” becomes a diff between two versions and a comparison of their outcomes.

The runtime harness

At call time, the harness loads the pinned version and builds the call’s runtime:

• The orchestrator holds call state and decides which sub-agent is active. Hand-offs between sub-agents carry structured state, not just conversation history.
• The tool layer gives every tool a typed contract, enforces each sub-agent’s allow-list, and applies timeouts and retries consistently. A sub-agent cannot call a tool it wasn’t given, whatever the model decides.
• The model router resolves a provider and model per sub-agent. We support several providers, including AWS Bedrock, so a sub-agent can use a fast model for intake and a stronger one for scheduling logic, and so a provider outage or a regression in one model doesn’t take everything down.

export async function buildCallRuntime(callId: string, phoneNumber: string) {
  const version = await registry.pinnedVersion(phoneNumber);   // immutable
  const config = await registry.load(version);
 
  const agents = mapValues(config.agents, (def, name) =>
    createAgent({
      name,
      instructions: compose(config.base, def.goal, config.clinic),
      tools: toolLayer.scoped(def.tools, { callId, clinic: config.clinic }),
      model: modelRouter.resolve(def.model),
      handsOffTo: def.hands_off_to ?? [],
    }),
  );
 
  return new Orchestrator({ callId, version, agents, entry: "intake" });
}

The visual builder

YAML is the source of truth, but most edits don’t come from engineers. A visual agent builder in the web app lets the team change instructions, tools and hand-offs, shows a diff against the published version, and saves through the same validation as everything else. It writes the same YAML; it doesn’t have its own format.

Measuring by version

Because every call carries its version:

• Analytics are keyed by version, so outcomes like booking rate and transfer rate can be compared across changes.
• A/B tests split traffic between two versions of the same agent and compare them scenario by scenario.
• The eval gate runs the simulation suite against a new version before it can be published, and blocks it on regressions.

What I learned

• Treat agent definitions like code: reviewable, versioned, validated, and deployed by pinning, not editing.
• Put shared behaviour in shared bases. The number of agents you can run well is limited by how many places a fix has to be made.
• Small agents with narrow tools beat one big agent with every tool.
• Record the version on every call. It’s the cheapest field in the system and the one I’d least want to lose.