Never a Meat Proxy: Human-in-the-Loop Discipline for Clinical AI

Every clinical AI deployment eventually comes down to one question: who checks the output before it matters? The demo never shows this part. It shows the model drafting a note or suggesting an order in seconds. What it does not show is the moment a tired clinician clicks approve without reading, and the model's confident mistake becomes a fact in someone's chart. That moment is where clinical AI is actually safe or not.

My rule is simple and I do not bend it. No AI output reaches a patient or a chart until a responsible human has read it, understood it, and verified it. Simon Willison has a sharp name for the alternative, borrowed from Niklas Gruhn: a meat proxy, a person who passively relays what the model said. In medicine that is not a style problem, it is a safety failure. Here are the principles I use to keep the human in the loop actually in the loop.

Willison, borrowing the term from Niklas Gruhn, describes the meat proxy: someone who takes AI output and relays it without reading, understanding, or validating it. His fix is to actually engage, read it, understand it, validate it, then put it in your own words as a certificate that you did. In clinical work that certificate is the whole point. A clinician who signs an AI-drafted note without reading it has not supervised the AI. They have laundered its output through a human name.

This is not hypothetical caution. In a study of trained experts making AI-assisted decisions under time pressure, researchers measured a 7 percent automation bias rate, where initially correct human judgments were overturned by erroneous AI advice. Seven percent sounds small until you multiply it across every AI-assisted decision in a health system. And pressure made it worse. The lesson is not to distrust every model, it is to design for the fact that a plausible, confident, wrong answer is exactly the one we are most likely to accept.

You cannot deeply verify everything, and pretending you will is how verification quietly becomes rubber-stamping. So I put the strongest human review where an error reaches a patient or a permanent record, and I let low-stakes outputs run lighter. A drafted internal summary and an order that changes treatment do not deserve the same scrutiny. Name the high-harm outputs explicitly, and make the human step there non-negotiable.

The failure I watch for most is the review step that exists on paper but not in practice. If the interface makes approving as effortless as breathing, people will approve on autopilot, especially when the model is usually right, which is what makes the occasional wrong one so dangerous. So I design friction where it matters: show the evidence, require the reviewer to engage with the specific claim, make them do something that only makes sense if they actually read it. The point is not to slow people down for its own sake. It is to make real review the path of least resistance.

In the end, a person puts their name on what reaches the patient, and that name has to mean something. The model can draft, suggest, and speed things up. It cannot be accountable. So the human in the loop is not there as a courtesy, they are there as the accountable party, and everything about the system should reinforce that they are responsible for what they approve. When the name on the chart is a human's, the verification behind it has to be too.

Human-in-the-loop is not a checkbox on a compliance form, it is the load-bearing safety mechanism in almost every clinical AI system I would trust. And it only works if the human is really there: reading, understanding, verifying, and willing to say no. Build the system so that is the easy path, measure whether it is actually happening, and never let the human become a rubber stamp with a pulse. The model can do a great deal. Owning the outcome is still ours.