A model that predicts the end of life can do harm even when it is accurate

Across medicine, doctors lean more and more on AI models that predict patient risk: sepsis, falls, death. The models often sit inside the electronic medical record, so their outputs slide easily into treatment. It is just as easy to take them for granted and stop checking what they measure.

James Deardorff is a geriatrician and assistant professor at the Division of Geriatrics at the University of California, San Francisco. He has built several predictive models for older adults, from mortality to the need for nursing home care. In geriatrics, he says, two things matter: how a model performs, including in subgroups such as older patients, and how its output is used.

An accurate model, a bad outcome

In September 2026 Deardorff published a commentary on a large analysis of Epic's proprietary end-of-life prediction model, published in JAMA Network Open. His argument is precise. A model can contribute to a bad outcome even if the model itself is accurate. If a prediction of one-year mortality risk opens a conversation about goals of care, the side effects are minimal. If the same output shapes decisions with bigger consequences, such as the order of qualification for transplantation, the impact can be serious.

What matters is not only the model's metric, such as AUC, but the decision pathway at the end of which the output is used. The model does not make the decision. A person who sees the number makes it. That is the place that needs designing, not training.

How to check it

The same problem, how to evaluate models that change faster than the studies of them, is described in a commentary in npj Digital Medicine from 17 September 2026 (Mahajan, Shah, Powell). The authors point out that the pace of model development, along with the variety of designs and ways of reporting prospective clinical studies, makes it hard to assess effectiveness and safety reliably. For a hospital this means that deploying a predictive model requires its own local monitoring of outcomes, because registry data from outside is not enough.

The practical checklist for clinical teams is short: in which subgroup of patients was the model validated, what decision does its output support, who is responsible for making that decision, and what happens when the prognosis is wrong. Without answers to those four questions, the number on the screen is just a number.

This is not a technical novelty. A model wired into the documentation system sees the data that staff enter as they work, and it can run in the background without asking permission for every read of the record. That is exactly why the line between decision support and surveillance of the patient is so thin here.