Medical AI privacy audits miss patient-level leaks, Nature paper finds

German researchers tested seven medical AI datasets made up of images, ECG records and general electronic health records. Membership inference attacks (MIAs) query a model to work out whether a specific datapoint sits in its training set. On individual patients they worked far better than aggregate privacy metrics suggest, The Register reported on 24 June.

The paper's conclusion lands awkwardly for anyone auditing medical AI under current rules. "The fact that MIAs can achieve near-perfect success rates for individual patients is not adequately captured by the standard evaluation protocol, which measures attack success in aggregate across records," the researchers wrote. Their argument: AI privacy audit reporting standards need to change.

Who is easiest to identify

Patients who are underrepresented in a training set are the easiest to flag. Race, insurance status, sex, the imaging protocol used and certain disease statuses can all act as outliers that sharpen an attack, according to the paper.

Moritz Knolle chairs AI in Healthcare and Medicine at the Technical University of Munich and led the paper. He described the practical stakes to The Register by email. "Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific," he said. Membership, he added, could reveal a dormant genetic condition such as Huntington's disease, depression, or attendance at a specialised treatment clinic. A leaked training set can expose conditions people have not disclosed anywhere else.

Two more findings make the picture worse. The larger the dataset, the easier it is to expose individual records. The paper says nobody had measured the size of that shift in patient-level risk in bigger models before.

What an attacker actually needs

The attack depends on a simple quirk: a medical AI tends to be more confident when the input already formed part of its training data. An attacker feeds in patient data, reads the confidence level, and infers membership from it.

"In our paper we show that an attacker with partial access can still successfully conduct MIAs," Knolle told The Register.

That matters because the earlier assumption was that a full patient record was needed. Knolle said an attacker would "simply need access to someone's blood test results, or part of these results" to infer inclusion, and that the datasets used in the study were anonymised. The target data, in other words, need not be.

Getting hold of that data is the harder part, but not prohibitively hard. Knolle pointed to the frequency of healthcare breaches. An attacker, he suggested, might gain unauthorised access to a general practitioner's database after a routine blood test.

Why the audit rules are the story

The paper does not argue that every successful MIA is a scandal. Knolle told The Register that many successful attacks represent a small or negligible privacy violation. He named the case of models trained on large, general populations where healthy and diseased individuals are both well represented. The problem is narrower and harder to regulate: specific cohorts, rare conditions, and any group that appears too rarely in the training data.

In April, a separate study in JAMA Network Open led by Harvard medical student Arya Rao tested 21 off-the-shelf AI models on 29 clinical vignettes. The models reached a correct final diagnosis 91 percent of the time, but failed at early differential diagnosis in more than 8 out of 10 cases, The Register reported on 15 April. Coauthor Dr. Marc Succi, a radiologist at Massachusetts General Hospital, said such systems "can project confidence without showing robust reasoning." The privacy paper exploits the same pattern: model confidence is not a reliable signal, and two very different contexts are treating it as one.

Regulators are moving, though the direction is unsettled. Recent headlines collected for context include a UK proposal for "L-plate" AI healthcare regulation, state-level rulemaking in the US, and guidance from Australia's TGA on AI medical devices. None of that addresses the specific gap Knolle's team identified. The gap is in how privacy is measured, not in whether a device is approved.

The researchers recommend differential privacy frameworks, which aim to give a mathematical guarantee that training data stays anonymous, and a rewrite of privacy audit standards so they assess individual-level risk rather than aggregate risk. A third option Knolle raised is blunter: compile training data so underrepresented groups appear more often. That reduces the outlier effect which makes them identifiable in the first place.

"I hope that the medical AI community will start to take privacy risks seriously and that risk mitigation techniques are used in situations where they are necessary," Knolle told The Register.