Patients in the training set are easy to pick out, and privacy audits do not catch it

Membership inference attacks (MIAs) ask a model whether a specific record was part of its training set. The Register reported the findings on 24 June. Discriminative medical AI models are especially exposed to this kind of attack. They are trained to classify inputs, and they grow measurably more confident about data they have already seen.

That confidence is the leak. The team analysed seven medical AI datasets covering images, ECG records and general electronic health records. Patients targeted by such attacks could be identified with what the paper describes as near-perfect success. Standard evaluation misses this, the authors say, because it measures attack success in aggregate across records rather than per person.

Outliers are the easiest to expose

Underrepresented groups in a training cohort are easier to finger than typical patients. Race, insurance status, sex, the imaging protocol used and certain disease statuses can all mark someone as an outlier.

Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific.

That is Moritz Knolle, who chairs AI in Healthcare and Medicine at the Technical University of Munich and led the paper, in an email exchange with The Register. He gave the example of a model whose membership list would reveal that someone has a dormant genetic condition such as Huntington's disease, or depression, or attended a specific specialised treatment clinic. Larger datasets made things worse, not better. The paper states that the magnitude of the change in patient-level risk in bigger models was previously unknown.

The attack also needs less than researchers once assumed. Knolle told The Register that an attacker normally needs access to a target data point, and that the paper shows partial access is enough. Blood test results, or part of them, would do. All datasets used in the study were anonymised, and the attacker does not need to know whose data they are testing.

Getting that data is the practical hurdle, though not a high one. Knolle said that because medical data is not always stored securely, unauthorised access to a general practitioner's database after a routine blood test is not unthinkable.

What the authors want changed

The paper's recommendations are procedural rather than technical fixes to the models themselves. The team wants privacy audit standards rewritten to assess individual-level risk instead of aggregate numbers. It points to differential privacy frameworks as a way to give mathematical guarantees that training data stays anonymous. Knolle also suggested training data could be compiled so underrepresented groups are better represented.

The stakes are not theoretical. A separate study published in JAMA Network Open in April and covered by The Register found that 21 off-the-shelf AI models failed at early differential diagnosis in more than 8 out of 10 cases, though leading models reached 91 percent accuracy on final diagnosis. Marc Succi, a radiologist at Massachusetts General Hospital and a coauthor, said such systems should not be trusted for patient-facing diagnostic reasoning without structured comprehensive human review.

Knolle framed the privacy problem with a caveat: there are many situations where a successful membership inference attack represents a small or negligible violation, namely when models are trained on large, general populations that include both healthy and diseased people. The difficulty is that the paper's own results show those are exactly the conditions under which audit protocols assume safety, and exactly the conditions the authors say are not being measured properly.