Log inSign Up
V. LazukaCTO
April 24, 2026
A 26-year-old woman pastes her blood work into a chatbot. It tells her the results "most likely" point to a pituitary tumor. She spends weeks terrified, waiting for an MRI. The scan comes back clean. There was nothing there.
A 63-year-old man describes his symptoms and recent labs to another chatbot. It reassures him. Suggests he escalate only if things get worse. He pushes for a catheterization anyway, on his own instinct. One of his arteries is 85% blocked. The kind of blockage doctors call a widowmaker.
Both of these cases were recently reported by The New York Times. Both people had done what millions of people now do every day, upload their medical records, lab results, scans, sometimes full oncology notes, into a general-purpose AI chatbot and ask what it all means.
Here is the thing almost nobody is talking about. The real danger is not what the AI says. The danger is what it leaves out.
A new study just put a number on it.
In December 2025, a team from Stanford, Harvard, and the ARISE AI Research Network published a benchmark called NOHARM. The name is a nod to primum non nocere, the oldest rule in medicine: first, do no harm.
They did something that had not really been done before. They took 100 real cases where a family doctor had written to a specialist asking for advice. Actual patients, actual questions, anonymised, pulled from Stanford Health Care's electronic consult system. Things like "my patient has unexplained bruising, what should I work up?" For each case, a panel of specialists wrote out every single reasonable action a doctor could take. Order this test. Don't order that one. Start this drug. Send to the emergency room. Reassure and watch.
4,249 possible actions across the 100 cases. Then 29 board-certified physicians, 23 of them specialists and subspecialists, rated every single one. Is this the right move, or the wrong one? And if a doctor did it when they shouldn't have, or failed to do it when they should have, how badly would the patient be hurt? Mild, moderate, or severe, using the World Health Organization's own definitions.
Then they ran 31 different AI models through the same 100 cases. GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro, DeepSeek, Grok, Llama, plus two specialised medical AI systems built for doctors (AMBOSS LiSA 1.0 and Glass Health 4.0).
Here is what they found.
The best-performing models, Gemini 2.5 Flash and AMBOSS LiSA 1.0, produced between 11.8 and 14.6 severe-harm recommendations per 100 cases. The worst models, the "mini" variants from OpenAI, produced around 40.
Translated into a number doctors actually use, "number needed to harm", it means that with the worst model, one in every 4 or 5 patients who used it was on track to be seriously hurt. With the best model, about one in 11.
That is not a theoretical risk. That is the floor. That is what the best publicly available AI, in December 2025, does on real patient cases when a trained specialist rates its output against the WHO's own harm scale.
Now here is where it gets uncomfortable.
77% of the severe harms were errors of omission. The AI did not tell someone to take the wrong drug. It failed to tell them to get the test that would have caught the cancer. It did not flag the kidney function that should have stopped a prescription. It did not escalate the chest pain that needed escalating.
This is the single most important finding in the study, and it is the exact opposite of what almost everyone worries about. People imagine the AI is going to tell them to do something dangerous. The study shows the AI is far more likely to quietly not mention the thing that would have saved their life.
And you, the patient, have no way to know. The AI sounds equally confident when it is complete and when it is missing something critical. There is no calibration signal. There is no warning light.
This is the finding that should make every "just use the best model" instinct break down.
The researchers checked whether safety performance correlated with any of the usual proxies for AI strength. Model size. Recency. Reasoning mode. Performance on standard benchmarks like MedQA, GPQA Diamond, LMArena. The correlations were weak at best. Safety showed only moderate correlation (r = 0.61 to 0.64) with general AI benchmarks. And it showed no correlation at all with completeness.
Gemini 2.5 Flash, a lighter, cheaper sibling of its flagship, outscored much larger models. GPT-5 came in at 58.3%, strong but not the top. The newest and biggest is not the safest.
Clinical safety is its own dimension. It is not something that comes for free with raw intelligence.
OpenAI's smaller models in particular were tuned to hedge, to qualify, to refuse when uncertain. On paper this feels safer. "Better to say nothing than to say something wrong."
Except in medicine, saying nothing is also a decision. And it is usually the wrong one. When a model refuses to recommend the test that would have caught the diagnosis, that refusal is itself the harm.
The study measured this directly. OpenAI's o3 mini scored highest on restraint (the least willing to recommend anything), and also the lowest on completeness (the most likely to miss a critical action the patient needed). One of those numbers sounds prudent. The other tells you what it actually costs.
Put all three findings together and the picture is clear. You cannot pick a safer AI for your medical questions by picking the newest model, the biggest model, or the one that seems most cautious. Those heuristics do not work. They were all measured, and they all failed.
One thing in the study reduced harm significantly. Multi-agent systems. Specifically, multiple AI models from different companies, passing work between each other, with later models prompted to catch what earlier models missed.
The single best configuration in the whole study combined Meta's Llama 4 Scout, Google's Gemini 2.5 Pro, and AMBOSS's medical system LiSA 1.0. One open-source, one proprietary, one specialised. None of them alone would have come close.
This is a structural finding, not a model-picking finding. It says: the safety of the system is a property of the architecture, not the intelligence of any one component. You cannot buy your way to clinical safety by buying a smarter model. You have to build a system where different models check each other's blind spots.
Three honest things, before we get to what we do.
First, a single general-purpose chatbot has no way to tell you when it is about to miss something important. The confidence in its voice is not correlated with the completeness of its answer. Ever.
Second, "I used the newest one" is not the safety story people think it is. The newest frontier models in the study were average on safety and average on completeness.
Third, the harm is rarely dramatic. It is a test not ordered, a referral not mentioned, a medication interaction not flagged. You will not know it happened. Neither will the chatbot.
This is not an argument against using AI for your health. It is an argument for knowing what you are actually getting when you do.
This is where BloodGPT comes in, so let us be direct about it.
We have been saying for two years that the system matters more than the model. The NOHARM study is, as far as we can tell, the most thorough evidence yet published that we were right.
Our approach has two halves, and they map exactly onto what the study found.
Where the task is structured, we do not rely on model intelligence at all. In a separate piece of work this year, the MedAgentBench benchmark run by Stanford, we hit 100% accuracy on structured EHR tasks using a lightweight model, while frontier models with much larger context and more reasoning power topped out at 98%. The reason is architectural. We separated data retrieval from computation.
Where the task is open-ended clinical reasoning, the kind of reasoning NOHARM measured, where no single model is safe enough, we run a pipeline of multiple frontier models checking each other's work. GPT-5.1 and Gemini 3 Pro sit at the top of that pipeline. They do not produce the answer directly. They review, cross-check, and flag omissions against a locked abnormality list that has to be addressed completely. Every abnormal value gets an interpretation. Nothing goes missing quietly, because the architecture does not allow it to.
That is the whole argument in one sentence. When the task is deterministic, we hand it to code, not to a language model. When the task requires judgment, we use multiple models from different companies, structured exactly the way the Stanford study showed reduces harm the most.
Not because we don't trust any one model. Because the evidence, now, is explicit: no one model is safe enough alone. That was always our bet. It is starting to look less like a bet and more like the only thing that holds up under real measurement.
If you paste your labs into a chatbot tonight, you will get an answer. It will sound confident. And somewhere in that answer, most of the time, will be something the model quietly did not say.
That is the part worth building for.
For the full technical write-up on how BloodGPT achieved 100% on Stanford's MedAgentBench, download the PDF here.
Link to NOHARM study · Link to BloodGPT Stanford's Test article
Want to see it in action?
bloodgpt.com
For developers: docs.bloodgpt.com
Talk to us: [email protected]
About BloodGPT
BloodGPT is a B2B healthcare AI platform that turns unstructured laboratory data into standardised FHIR R4 records with clinical interpretation. We operate in Israel, Nigeria, Poland, the UK, and Greece, with enterprise deployments including white-label, API, and on-premise options.
Study:
Wu, Nateghi Haredasht, et al. "First, do NOHARM: towards clinically safe large language models." arXiv:2512.01241, December 2025.
Interactive leaderboard: bench.arise-ai.org.
Transform how you deliver lab results to patients while enhancing your practice's efficiency and value. Try BloodGPT or contact us for a personalized demo
Book a Demo
