Medical diagnosis AIs leak who trained them, and the rules don't catch it

The finding, covered by The Register on 24 June, comes from a team led by Moritz Knolle, who chairs AI in Healthcare and Medicine at the Technical University of Munich. The paper tests membership inference attacks (MIAs) against seven medical AI datasets built from images, ECG records and general electronic health records. The models are discriminative: they classify inputs and make predictions based on what they were trained on.

An MIA exploits a simple quirk. A model tends to be more confident about inputs it has seen before. Feed it a record, read the confidence score, and you can infer whether that record was in the training set. Knolle says an attacker running an MIA does not need to know who the data belongs to, and all the datasets used in the study were anonymized.

Underrepresented patients are easier to identify

The research found that patients in a dataset are generally easy to single out, and that those underrepresented in training data are easier still. Race, insurance status, sex, the imaging protocol used and certain disease statuses can all act as outliers that make an individual stand out.

Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific. You could imagine ... scenarios where membership in a training dataset reveals that someone has a dormant genetic condition such as Huntington's disease, depression, or attended a specific, specialised treatment clinic.

That quote is from Knolle, speaking to The Register by email. The implication is not abstract: knowing that a person's data sits inside a specialist cohort can reveal a diagnosis they never chose to disclose, or feed discrimination in insurance, employment or elsewhere.

Scale makes it worse, not better. The paper found that the larger the dataset, the easier it becomes to expose records, and that "the magnitude of this change in patient-level risk was previously unknown" in larger models. That runs against a common assumption that bigger training sets dilute any individual's exposure.

The audit numbers miss the point

This is where the regulatory argument starts. The researchers say near-perfect success against individual patients "is not adequately captured by the standard evaluation protocol, which measures attack success in aggregate across records." An audit can report a low average attack success rate while individual patients remain trivially identifiable. The team's conclusion is that reporting standards for AI privacy audits need to change.

Knolle's recommendations cover two tracks. The first is technical: differential privacy frameworks, which are designed to give a mathematical guarantee that training data stays anonymous. The second is procedural, moving audits from aggregate privacy risk to individual-level risk. He also suggests a blunter fix, compiling medical AI training data so underrepresented groups are better represented in the first place. "I hope that the medical AI community will start to take privacy risks seriously and that risk mitigation techniques are used in situations where they are necessary," he told The Register.

There is a real precondition for any of this. To run an MIA, an attacker needs at least some data belonging to the target. Knolle said the paper shows a full patient record is not required, contrary to what was previously believed: "In our paper we show that an attacker with partial access can still successafully conduct MIAs." He gave the example of blood test results, or part of them, being enough to infer inclusion, and pointed at the routine exposure of healthcare data in breaches. "Given that medical data is not always securely stored it is not unthinkable that an attacker could get access, for example, by gaining unauthorized access to the database of your general practitioner after they performed a routine blood test," he said.

Knolle was careful not to overstate the harm in every case. "There are many situations where a successful MIA represents a small or negligible privacy violation," he noted, pointing to models trained on large, general populations where both healthy and diseased individuals are represented in sufficient numbers. The risk concentrates where cohorts are narrow.

The diagnostic models themselves are weak too

Privacy is only one of two problems visible in the same corner of medical AI. Research published in JAMA Network Open in April, and reported by The Register on 15 April, tested 21 off-the-shelf AI models across 29 standardized clinical vignettes. The models were right on final diagnosis 91 percent of the time when handed a full portfolio of information. On early differential diagnosis, the stage where clinicians are still ruling conditions in and out, they failed in more than 8 out of 10 cases.

The study was led by Harvard medical student Arya Rao. "Every model we tested failed on the vast majority of cases," Rao told The Register. "That's the stage where uncertainty matters most, and it's where these systems are weakest." Coauthor Dr. Marc Succi, a radiologist at Massachusetts General Hospital, said the models "can project confidence without showing strong reasoning, especially around differential diagnosis," and warned that the confidence can inflame anxiety in patients already worried about their health. Rao also noted that the strict failure metric is harsh: measured as raw accuracy, models scored between 63 and 78 percent, often partially correct rather than wholly wrong.

Put the two papers side by side and the regulatory gap widens. The privacy paper says audits measure the wrong thing, at the wrong level, and that vendors need an incentive to secure training data. The diagnostic paper says the marketing of these systems as frontline agents "risks fostering false confidence precisely where they are least reliable," and that "LLMs cannot yet be trusted in frontline decision-making." In both cases the failure is one that aggregate benchmarks and product claims can hide.

What the researchers are asking for is narrower than a ban. Change how privacy audits report risk, so individual patients count. Use differential privacy where cohorts are sensitive. Balance training data so outliers are not outliers. And treat the models as tools that need human review rather than as gatekeepers. None of that requires new science. It requires evaluation regimes that measure the thing that can actually hurt a patient.