Clinical AI Regulation Is Being Written in Rooms You Are Not In
The rules for clinical AI are being written now, often out of public view. Here is how I build and document so I am ready whatever they say.
Photo by Ben Kelsey on Pexels
The rules for clinical AI are being shaped right now, often in meetings the public never hears about. In my experience you cannot wait for the final rulebook to build, because it may be years away and it is being influenced by whoever shows up. So I build for the rules I expect, document everything, and stay close enough to the process to adapt when the lines finally get drawn.
In July, according to STAT, federal regulators quietly invited ten companies to an unannounced clinical AI demo day at the FDA's White Oak campus. Microsoft, Amazon's One Medical, Anthropic, and a handful of health-AI startups put their AI doctor tools in front of the FDA and CMS, and got to help shape how this technology is regulated and paid for. I do not think there was anything sinister about it. But it is a clean reminder of how the rules for clinical AI actually get made: not in a finished rulebook handed down from on high, but in rooms, over time, influenced by whoever is present.
So here is my stance for anyone building clinical AI. You cannot wait for the final rules, because they are years away and still being written. And you cannot ignore them, because they will eventually decide what you are allowed to ship and whether anyone pays for it. What you can do is build for the rules you expect, document everything as if an audit is coming, and stay close enough to the process to adapt. That is the honest middle path between paralysis and recklessness.
How the rules actually get made
Clinical AI is not regulated by a single moment of legislation. It is shaped in a slow loop, and understanding that loop tells you where to put your effort. Regulators get firsthand exposure, often through exactly the kind of demo day STAT described. Companies and researchers give informal input. Draft guidance appears. Then, eventually, something firmer. Each step influences the next, and the earliest steps, the ones with the least paper trail, are where a lot of the direction is set. If you are only watching for the final rule, you are watching the last and least changeable step.
Build for the rules you expect
Waiting for certainty is itself a decision, and usually the wrong one, so I build against the rules I am fairly sure are coming. Some things are safe bets no matter how the specifics land. You will need to show what your model was trained and validated on. You will need a human accountable for clinical decisions. You will need to monitor performance after deployment, not just at launch. You will need to explain, in plain terms, what the tool does and does not do. I build all of that in now, because I have never seen a plausible version of the rules that does not require it. The table below is how I sort what to build ahead versus what to hold.
Document like the audit is coming
The cheapest insurance against uncertain rules is a paper trail you build as you go, not one you reconstruct under pressure. When the rules do firm up, the teams that suffer are the ones piecing together history: what data did we use, who signed off, how did the model behave last quarter. So I document as I build. Model cards, validation results, decision logs, monitoring dashboards. The KLAS and UPMC governance research, where a dedicated validation environment and clear governance are treated as the norm, is a good template, and it is also where most organizations still fall short. The paper trail is boring until the day it is the only thing standing between you and pulling a product.
The rooms where clinical AI gets shaped will not always include you, and that is fine. What you control is whether your product is ready for the rules those rooms produce. Build for the requirements that show up in every plausible future, keep the receipts, and stay close enough to the conversation to move when it does. That beats waiting, and it beats pretending the rules will never come.
- Clinical AI rules are being shaped now, often in unannounced meetings, not in a finished rulebook.
- Do not wait for final rules, and do not ignore them; build for the requirements every plausible version will include.
- Safe bets: show training and validation data, keep a human accountable, monitor after deployment, explain the tool plainly.
- Treat certification specifics and payment rules as watch items, not roadmap bets.
- Document as you build; a paper trail you reconstruct under pressure is a paper trail you lose.
- Stay close to the process so you can adapt when the lines are finally drawn.
Frequently asked
Who is currently shaping clinical AI regulation?
In the US, agencies like the FDA and CMS, informed by industry and researchers through pilots and meetings such as the July 2026 demo day STAT reported, alongside standards bodies.
Should I wait for clear rules before building clinical AI?
No. The rules are years away and still forming. Build for the requirements every plausible version will include, and design so specifics are easy to add later.
What requirements are safe to assume?
Transparency about training and validation data, a human accountable for clinical decisions, post-deployment monitoring, and a plain explanation of the tool's scope.
How do I prepare for an uncertain audit?
Document as you build: model cards, validation results, decision logs, and monitoring. Reconstructing this later is where teams fail.
Why were the closed-door meetings a concern?
Because direction gets set early, with little paper trail, and only the companies in the room get to influence it, per STAT's reporting.
Sources
Naveen Kumar
Healthcare engineering and product executive in Pittsburgh. 15+ years building AI-first patient access, a decade at Treatspace.