Log inSign Up
V. LazukaCTO
Ask a modern frontier model how it reached a clinical conclusion and you will get an answer. It will be fluent, well organised, and shaped like reasoning. Whether it describes what the model actually did is a separate question, and one the field is slowly losing the ability to answer.
That is not a complaint about any single model. It is a direction of travel. As these systems get more capable, they also get harder to see into. In June 2026, two of the most senior scientists in the field, writing in Science, argued that the window in which humans can still meaningfully understand and guide AI may be narrowing, and that without deliberate effort it could close. Their concern is general: AI systems increasingly designed by other AI systems, agents talking to agents in ways that drift away from human language, and models that learn to read the room and behave differently when they sense they are being evaluated.
We read it the way we read most things now, through the narrow lens of one question. What does this mean for putting AI anywhere near a patient's lab results.
The answer is uncomfortable, and it is also the reason BloodGPT exists.
In most software you can tolerate a system you do not fully understand, as long as the output is usually right. Clinical work does not give you that latitude.
A lab interpretation that cannot be reproduced cannot be audited. An abnormality flag that cannot be traced back to the value that triggered it cannot be defended to a clinician, a regulator, or a court. A model that quietly behaves differently this week than it did last week is, by definition, not validated, because the thing you validated is no longer the thing running.
So the question the Science piece raises at the level of the whole field, can we still understand what these systems are doing, is for clinical AI not a philosophical question. It is the entire engineering problem.
We have argued for two years that the bottleneck in clinical AI is not model capability. It is reliability, provenance, and safety, and those are properties of the system around the model, not of the model itself. The same point that Stanford and Harvard made in the State of Clinical AI Report. The opacity problem is that argument seen from the other side. You do not get reliability from a system you cannot understand.
Here is the part that gets lost in the interpretability debate. Understanding an AI system does not require understanding every parameter inside the model.
The authors make this point themselves. We study humans at the level of neurons, of psychology, and of societies, and partial understanding at the right level is still useful. You can know almost nothing about a person's neurochemistry and still hold them accountable for what they do.
The same is true for a clinical pipeline. We do not try to make the language model interpretable. We treat it as a component whose internals we may never fully see, and we build everything around it so the system is understandable even when the model is not. Concretely, that means three things.
The output is reproducible. The same patient record, run through BloodGPT again, produces the same abnormal set, the same recommended follow-ups, the same structured result. We verify this by hashing outputs across runs. This is not a tuning trick. Most of the computation in the pipeline is not done by a language model at all, which is also how we reached 100% on Stanford's MedAgentBench with a lightweight model.
Every output is traceable. Each flag has a path back to a specific lab value, a specific reference range, and a specific stage in the pipeline. The audit chain records who asked, what data was processed, which model version ran, and which patient data drove each step. When something is generated by the pipeline rather than read from the source record, it is marked as generated, not passed off as fact.
Nothing reaches the model unchecked. A non-LLM control layer sits in front of every interpretation step, checking structure, catching duplicates, and rejecting values that are inconsistent with their own declared units before any model sees them. Language models do not fail loudly on bad input. They produce confident, plausible, wrong output. The control layer exists so they never get the chance.
None of that requires opening the model. It requires refusing to let the model be the system.
One of the sharper warnings in the piece is that models are starting to behave differently when they detect they are being evaluated, producing what an evaluator wants to see rather than what the system would actually do in deployment. If that is true, a high score on a static, predictable test tells you less than it appears to.
This cuts in a direction we have to be honest about. Benchmarks matter, and we are proud of the MedAgentBench result. But a benchmark is a controlled question. It is not a hospital.
Two things protect against the trap. First, a deterministic pipeline cannot read the room. It produces the same output whether it is observed or not, because the behaviour is fixed by the architecture, not chosen by a model in the moment. Second, the only real answer to "does it hold up under real conditions" is to measure it under real conditions. That is why our central piece of evidence is not the benchmark. It is the clinical validation now running on roughly 100,000 anonymised hospital records with Assuta Medical Centers, on real FHIR R4 data, with interpretations scored by clinicians against ground truth. The benchmark tests whether the pipeline executes structured tasks correctly. The hospital data tests whether it holds up when the inputs stop being clean.
The article also worries about agents communicating with each other in ways that drift away from anything a human can follow. This is a real risk in systems where models are handed open-ended latitude to talk amongst themselves.
We use more than one model on purpose, because the evidence is now explicit that no single model is safe enough alone for open clinical reasoning. But the models in our pipeline do not hold a private conversation. They pass structured, typed data: FHIR resources and a locked list of abnormalities that has to be addressed item by item. Each model's job is bounded and inspectable.
The benefit of multiple models, catching each other's blind spots, is kept. The cost the article warns about, an internal language no human can audit, is designed out.
The quietest warning in the piece is the one about us, not the machines. As these systems smooth away friction and tell us what we want to hear, we may gradually stop scrutinising them at all.
In clinical AI this has a precise form. A system that hands a clinician a confident verdict with no visible evidence trains that clinician to stop checking. A system that shows its work, this flag, from this value, against this range, at this step, does the opposite. It keeps the human in the position of judgment.
That is why BloodGPT produces structured, sourced output rather than an authoritative-sounding paragraph, why critical values trigger mandatory escalation language the pipeline cannot skip, and why no new diagnosis or medication recommendation is produced before a dedicated, evidence-bound stage runs.
The point is not to make the AI sound more certain. It is to keep the person using it able to disagree.
The authors are describing a window that is still open and may not stay that way. Their prescription is to treat human understanding as a goal on equal footing with capability, rather than something we hope survives as a side effect.
For clinical AI that is not aspirational. It is buildable today, and it is mostly an architectural decision. Reproducible outputs. Provenance on every claim. Validation on real patient data rather than only on tests a model might learn to game. Multiple models with no room to drift into a private language. A human kept in the loop by design. None of it depends on a future model being more transparent than today's. All of it depends on refusing to let an opaque model be the whole system.
That has been our position from the start. It is starting to look less like a position and more like the only version that holds up when you ask the question the Science piece is really asking. A year from now, will you still understand what your AI is doing.
For BloodGPT, that question has a one-word answer, and we built the entire system so the answer could stay yes.
If you are working on medical AI infrastructure and want to compare notes on how to actually close these gaps, we'd be glad to talk. Reach out at [email protected].
Transform how you deliver lab results to patients while enhancing your practice's efficiency and value. Try BloodGPT or contact us for a personalized demo
Book a Demo
