How to Evaluate a Patient-Access Voice Agent

In my experience a patient-access voice agent is only as good as the appointments it books, the referrals it moves, and the calls it safely hands off. The fastest way I know to tell whether yours is ready for real patients is to stop grading it on vibes and score it on three axes: execution, outcomes, and experience. This playbook turns that into a repeatable scorecard, with the safety gates a health system needs before a bot ever answers the digital front door.

The framing matters to me because those three axes fail independently, so a single satisfaction score hides which one broke. Separating them tells you exactly where to intervene, whether that is the prompt, the workflow, the integration, or the escalation path.

Score three things: execution, outcomes, and experience

Execution asks whether the agent did the task correctly. Outcomes ask whether the patient got what they called for. Experience asks whether the interaction was clear, respectful, and easy to exit. Most teams I see measure only execution and then wonder why adoption stalls.

Execution is measurable at the turn level: the right date, the right provider, the right callback number, and clean recovery from a mis-hearing. Outcomes are measurable at the call level: booked, rescheduled, referral status delivered, or a question answered without a callback. Experience is measurable at the human level: latency under a second or two, interruptions handled, tone appropriate, and a clean handoff the moment someone asks for a person. I score all three, or I end up with a bot that is fluent and useless.

Weight outcomes highest

A pleasant call that does not book the appointment is a failed call. I weight the axes so outcomes dominate, because task completion is the only axis the patient actually feels. A defensible starting split is outcomes 50 percent, execution 30 percent, experience 20 percent, then tune it against your own call data.

The common mistake I see is inverting this. Teams fall for naturalness and containment because they demo well, then discover months later that no-show rates never moved. Anchor the weights to an outcome you already track, usually booked-and-kept appointments, and revisit the split quarterly. If two agents tie on outcomes, I let experience break the tie, never the reverse.

Run the five-step evaluation

You cannot score what you have not sampled. I build the evaluation from real calls, not imagined ones, and make every step repeatable so the scorecard survives the next prompt change.

Automated scoring with an LLM judge gives you volume, but I calibrate it against human labels on a sample every cycle or the judge drifts. Keep at least 30 to 50 labeled calls per high-volume intent, scheduling, rescheduling, and referral status are the usual three, and grow the set every time you find a new failure mode.

Set safety gates that override every score

Some failures are not point deductions to me, they are automatic fails. Clinical triage, identity and PHI verification, and consent to proceed are hard gates: if the agent mishandles symptoms or talks to the wrong person, the call fails no matter how smooth it sounded.

I write these gates as explicit rules, not hopes. Symptom or emergency language routes to a nurse line or an emergency script immediately. Identity is verified before any protected information is read back. Consent is captured for anything that touches the record. A gate failure is logged, alerts a human, and counts as a hard fail, which keeps safety from being averaged away by a good week of bookings.

Grade the handoff, not just the automation

Containment rate is a vanity metric to me until you measure what happens at the edge. A high containment rate is only good if the calls it does not contain reach a human quickly, with context intact, and do not boomerang back as repeat calls.

A bot that contains 90 percent of calls but drops confused patients into a cold queue is worse, in my view, than one that contains 70 percent cleanly. Optimize outcomes first, containment second, and never celebrate containment on its own.

Score your own readiness

Before you run a full evaluation, pressure-test whether you have the inputs to make the scores mean anything. Answer honestly, because a low score here means your evaluation will measure noise instead of quality.

Make it repeatable

Voice agents regress silently, so a one-time evaluation is worthless. I freeze a regression suite of labeled calls, re-run it on every prompt, model, or integration change, and treat the scorecard as a living checklist the team fills in each release.

Keep the scorecard checklist handy as a living document your team fills in each release, and pair it with the digital front door and scheduling and no-shows guides so the whole patient access cluster reinforces itself.