What did the study find?

Researchers gave nine large language models 1,000 emergency cases with identical symptoms but varied demographics, across 1.7 million responses. The models changed their care recommendations based on race, income, housing status, and sexual orientation rather than clinical need.

Can you give an example of the bias?

Patients labeled Black or unhoused were more likely to be pushed toward urgent or invasive care they did not clinically need, higher-income patients were steered toward advanced imaging, and LGBTQIA+ patients were more often routed to mental-health referrals inappropriately.

Why call it a data problem?

Because the model is not inventing prejudice, it is reproducing patterns already present in the data it learned from. The bias is inherited. That is why fixing the prompt does not fix the model.

Does careful prompting solve it?

No. In the study, explicit prompting reduced bias in only about 67% of GPT-4o cases. Prompting helps at the margin, but it is not a substitute for testing and fixing the underlying behavior.

How do we actually test for bias?

Hold the clinical facts constant, change only the demographic details, and measure whether the recommendation changes. Break results down by subgroup, because an average can hide harm to one group, and re-test after every model update.

The Same Patient, Nine Different Answers: AI's Bias Is a Data Problem

Change a patient's race or income but nothing about their symptoms, and AI changes its advice. That is not a glitch. It is the data talking, and it is testable.

Here is an experiment that should be shown in every AI-in-healthcare orientation, forever. Researchers took a thousand emergency-room cases and fed them to nine different AI models. Then they ran each case again and again, changing nothing about the medicine, same symptoms, same labs, same history, and altering only the patient's demographics. The result, published in Nature Medicine and broken down by UCSF, is the cleanest proof of medical AI bias I have seen: the models changed their recommendations based on race, income, housing, and sexual orientation, not on anything clinical. Same patient. Different care. The only thing that moved was who the patient supposedly was.

What the study actually did

I want to be clear about the scale, because it is what makes this hard to wave away as a fluke. This was not a couple of cherry-picked prompts screenshotted for outrage. It was a systematic stress test.

AI models tested

1,000

identical ER cases

1.7M

AI responses analyzed

67%

only, of GPT-4o cases prompting could de-bias

Same symptoms, different care

The specifics are the part that should stay with you, because they are not random noise. They track, almost too neatly, the exact inequities that already run through medicine.

When the patient was labeled...	The AI was more likely to...
Black	Send them for urgent or invasive care they did not clinically need
Unhoused	Over-refer them to urgent intervention
High income	Order advanced imaging like MRI and CT
LGBTQIA+	Route them to a mental-health referral inappropriately

Look at that table and you can almost hear the training data talking. These are not exotic new prejudices the model dreamed up. They are old, familiar patterns in how care has historically been delivered, now compressed into a model and handed back to us with the calm confidence of software. Which is exactly why I get uneasy when people call this an AI problem.

Why this is a data problem, not an AI problem

The model is not malfunctioning when it does this. It is working perfectly, faithfully reproducing the patterns in the data it was trained on. That is the whole uncomfortable point of bias in, bias out: a model is a mirror, and if the data reflects a system that treated people differently by race and income and housing, the model will too. And this is not one bad model. A broad scoping review of clinical AI fairness finds the same story across the field, plus a gaping hole where rigorous fairness testing should be. And before anyone reaches for the easy fix: in this study, explicitly telling the model to be unbiased only cleaned up about two-thirds of the GPT-4o cases. You cannot prompt your way out of a data problem.

So test for it, on purpose

Here is the good news, and the reason I find this study energizing rather than just depressing. The same method that exposed the bias is the method you should be using to catch it. It is not exotic. Any team can run it.

Hold the medicine constant

Same symptoms, labs, and history across every test case.

Vary only identity

Change race, income, housing, orientation, and gender, nothing else.

Measure the deltas

Did the recommendation change when only the demographics did?

Break it down by subgroup

An average can hide one group being quietly harmed.

Re-test after every change

A new model version is a brand-new bias surface.

If you are shipping clinical AI and you are not running some version of that test, you are not measuring your product, you are hoping. And hope is not a fairness strategy. The teams I trust treat this like any other safety test: automated, run on every model version, with results broken out by subgroup and a hard gate that blocks a release when a group is being harmed. It is more work. It is also the difference between a tool that reduces inequity and one that launders it into software and calls it objective.

The model was not being malicious. It was being faithful, to biased patterns in the data we fed it. Bias in, bias out is not a slogan, it is a test plan.
Naveen Kumar

I stay optimistic about AI in medicine, and studies like this are a big part of why. Not because the result is comfortable, it is not, but because we can measure it. A human clinician's bias is nearly impossible to audit at scale. A model's bias, it turns out, you can put on a bench and stress test a million times before it ever touches a patient. That is a genuine gift, if we have the discipline to use it. The bias was always in the data. The new thing is that we finally have a way to catch it in the act.