Medical AI can name the patients it trained on, researchers find

The attack is called a membership inference attack, or MIA. The Register reported on 24 June that the team probed seven medical AI datasets, covering images, ECG records and general electronic health records. Individual patients targeted this way can be identified with what the researchers call near-perfect attack success. The paper says the standard evaluation protocol misses that figure, because the protocol measures attack success in aggregate across records.

Medical AIs are more confident when the input data already sits in their training set. An attacker feeds a model patient data, reads the confidence level, and concludes the patient was part of the training cohort.

Technical University of Munich researcher Moritz Knolle, the paper's lead author, told The Register that an attacker does not need to know who the data belongs to. "In fact, all the dataset we use in our study were anonymized," he said. Partial access is enough. "The attacker would simply need access to someone's blood test results, or part of these results."

Outliers are the easiest to finger

The pattern of who gets exposed is uncomfortable. Underrepresented groups in medical training data are easier to identify than patients whose records blend in. The paper lists race, insurance status, sex, the imaging protocol used and certain disease statuses as categories that can function as outliers. Knolle's summary to The Register: "Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific."

He gave the example of a training cohort that reveals a dormant genetic condition such as Huntington's disease, or depression, or attendance at a specialised treatment clinic. Membership itself leaks the diagnosis. The paper also found that the larger the dataset, the easier records are to expose. It also found that the size of this shift in patient-level risk in larger models was not previously known.

The attacker needs some medical data about the people they want to identify. Healthcare breaches are frequent enough that this is not a high bar. Knolle raised the scenario of unauthorised access to a general practitioner's database after a routine blood test.

His stated aim is narrow: that the medical AI community takes privacy risks seriously and applies mitigation where it is needed. The paper recommends differential privacy frameworks, which are designed to give a mathematical guarantee that training data stays anonymous, and a rewrite of privacy audit standards to measure individual-level risk instead of aggregate risk. Compiling training data so underrepresented groups are better represented is another option he raised.

Regulators are still catching up on the basics

This lands in a regulatory picture that is moving, unevenly. IntuitionLabs, in a report updated on 18 August, puts the FDA's count of authorised AI and machine learning devices at roughly 950 by mid-2024, with about 100 new authorisations a year. Market analysts cited in the same report valued AI-enabled medical devices at 13.7 billion dollars in 2024 and projected more than 255 billion dollars by 2033. Radiology dominates authorisations, with cardiology, neurology, anesthesiology and ophthalmology expanding.

The same report notes that systematic reviews find only a tiny fraction of authorised AI devices are supported by randomised trials or patient outcome data. It also notes that device malfunctions have been linked to injuries and, in one reported case, a death. It cites an ICU triage tool that under-identified Black patients for extra care, and colonoscopy studies in which doctors' detection rates fell when they leaned on AI.

On privacy specifically, the paper's demand for individual-level auditing runs into a system built for aggregate reporting. FDA decision summaries often omit critical efficacy and safety details, according to the IntuitionLabs review, and global adherence to standards is uneven. The EU's AI Act, in effect since 2024, labels many healthcare AI systems high risk and adds compliance work on top of the Medical Device Regulation. The WHO published recommendations in 2023 covering transparency, data quality and lifecycle oversight.

MD+DI published an interview with FDA medical device attorney Suzanne Levy Friedman, who said the last count she had was over a thousand AI-enabled devices authorised by the FDA. Not one has a continually learning model built in, she said. Some of her newer clients are shocked by that. She is not.

"With all the excitement of innovation, people forget that first and foremost, FDA is a public health agency," Friedman told MD+DI. "They're not trying to be difficult, they're trying to make sure patients aren't harmed and clinicians have the right information and aren't inadvertently tricked into relying on things that might not be validated."

Friedman flagged two gaps that predate the privacy paper. Software iterates faster than the framework designed for hardware, and real-world performance drift can hit validated products once the patient mix shifts, as happened during COVID. Algorithmic bias sits alongside it, and FDA asks manufacturers for granular detail about their algorithms, which some find excessive.

The clinical evidence problem next door

Privacy is not the only weak point in the evidence base. The Register reported on 15 April on a JAMA Network Open study, led by Harvard medical student Arya Rao, that tested 21 off-the-shelf AI models against 29 standardised clinical vignettes. The models were correct 91 percent of the time on final diagnosis when handed a full portfolio of information. On early differential diagnosis, the stage where clinicians rule conditions in and out under uncertainty, the failure rate exceeded 80 percent.

"Every model we tested failed on the vast majority of cases," Rao told The Register. "That's the stage where uncertainty matters most, and it's where these systems are weakest."

Coauthor Marc Succi, a radiologist at Massachusetts General Hospital, warned that confidence without reasoning is its own hazard, particularly for anxious patients. He also argued that strong final-diagnosis scores can create a misleading sense of safety, because real clinical reasoning starts earlier, when ambiguity is highest. Rao noted that the stricter failure metric is not the only way to read the data: raw accuracy ranged from 63 to 78 percent, meaning models were often partially right.

The two papers point in the same direction from different angles. One shows that the training data behind a diagnostic model can be interrogated to identify the patients inside it. The other shows that the same models, asked to reason the way clinicians do, fall down at the point where the stakes are highest. Both arrived while the audit standards meant to catch such problems are still being drafted.

Knolle's caveat is worth keeping. He told The Register there are many situations where a successful MIA represents a small or negligible privacy violation, namely models trained on large, general populations that include both healthy and diseased individuals. The risk concentrates where cohorts are narrow, and narrow cohorts are exactly what specialist medical AI tends to need.