AI Diagnosis Under Scrutiny: Patients Reveal Training Data, LLMs Fail in 80% of Cases

Two lines of research published in 2026 paint a troubling picture of AI medical diagnosis. One shows that diagnostic models leak the identities of patients whose records were used to train them. The other shows that large language models, the same tools people query about their symptoms, fail at the early stages of clinical reasoning in more than 80 percent of cases. Both were reported by The Register, which covered the papers as they appeared.

The privacy problem comes from a Nature paper published on Wednesday 24 June 2026, according to The Register. German researchers tested seven medical AI datasets containing images, ECG records and general electronic health records. They found that discriminative models, the kind that classify data and make predictions, are especially vulnerable to membership inference attacks (MIAs). These attacks query a model to determine whether a specific data point was part of its training set. The team reported that individual patients can be identified with "near-perfect attack success." That success rate, they argue, is not captured by standard evaluation protocols, which measure attack success in aggregate across records. The researchers concluded that reporting standards for AI privacy audits need to change.

Underrepresented patients are easier to identify

Patients who are underrepresented in medical AI training data are even easier to identify than those whose data does not stand out, according to the same paper. The Register listed race, insurance status, sex, imaging protocol and certain disease statuses as categories that can function as outliers, making individuals easier to finger.

"Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific," Moritz Knolle, chair of AI in Healthcare and Medicine at the Technical University of Munich and lead author of the paper, told The Register in an email conversation. He described scenarios where membership in a training dataset could reveal that someone has a dormant genetic condition such as Huntington's disease, depression, or attended a specific specialized treatment clinic.

The attack itself relies on a simple property of machine learning models: they tend to be more confident when the input data was part of their training set. An attacker with partial patient data can feed it to the model, check the confidence level, and infer whether that patient was included. Knolle said the paper showed that an attacker does not need full access to a patient data point, in contrast to what was previously believed. "In our paper we show that an attacker with partial access can still successfully conduct MIAs," he told The Register.

"I hope that the medical AI community will start to take privacy risks seriously and that risk mitigation techniques are used in situations where they are necessary." Moritz Knolle, Technical University of Munich

The researchers recommend using differential privacy frameworks designed to mathematically guarantee that training data remains anonymous. They also want privacy audit standards to consider individual-level risk rather than just aggregate numbers. Another option, Knolle said, is to compile training data so that underrepresented groups are better represented. "There are many situations where a successful MIA represents a small or negligible privacy violation," he noted, referring to models trained on large general populations that include both healthy and diseased individuals.

Language models fail early, look confident

Separately, research published in JAMA Network Open in April 2026 found that 21 leading off-the-shelf AI models failed at early differential diagnosis in more than 8 out of 10 cases. The study, covered by The Register on 15 April 2026, tested the models on 29 standardized clinical vignettes. The models did well when given a full portfolio of medical information and asked for a final diagnosis, with leading models correct 91 percent of the time. But early differential diagnosis, where clinicians rule out conditions while weighing possibilities, was where the failure rate exceeded 80 percent.

"Every model we tested failed on the vast majority of cases," Arya Rao, a Harvard medical student who led the research, told The Register in an email. "That's the stage where uncertainty matters most, and it's where these systems are weakest."

Rao noted that failure did not always mean a completely wrong answer. Measured by raw accuracy as a proportion correct, the models ranged from 63 to 78 percent. But the team argued that the stricter failure metric still matters, especially because AI tools are marketed as frontline agents meant to narrow down diagnoses. "Marketing LLMs as diagnostic agents risks fostering false confidence precisely where they are least reliable," the researchers wrote.

Dr. Marc Succi, a radiologist at Massachusetts General Hospital and coauthor of the paper, told The Register that higher success rates on final diagnosis should not be reassuring. "Real clinical reasoning starts earlier, when ambiguity is highest, and that is exactly where they remain weakest," he said. He warned that a wrong differential can lead to delays in care, unnecessary procedures, high costs and more.

Both papers land in a regulatory environment that is still catching up. As of mid-2024 the US Food and Drug Administration had authorized around 950 AI-enabled medical devices, according to an industry analysis published by IntuitionLabs. The same analysis noted that only a tiny fraction of authorized AI devices are supported by randomized trials or patient-outcome data, and that the FDA issued its Predetermined Change Control Plans guidance in December 2024.

An interview published by MD+DI with FDA medical device attorney Suzanne Levy Friedman noted that despite more than a thousand FDA-authorized AI-enabled devices, none has a continually learning model built in. Friedman said the agency is working on a path toward that, but that its primary mission as a public health agency means it has to weigh innovation against the risk of clinicians relying on tools that may not be validated.

The two research papers suggest that validation is not just about whether a model gets the final answer right. It is also about whether it keeps a patient's identity secret, and whether it can handle uncertainty before a diagnosis is clear.