Medical AI has two problems at once: it guesses wrong early and leaks who trained it

On 15 April 2026, The Register reported on a JAMA Network Open study that tested 21 off-the-shelf AI models against 29 standardized clinical vignettes. Given a full portfolio of medical information and asked for a final diagnosis, the models scored 91 percent. Early differential diagnosis, the stage where clinicians rule conditions in and out, failed in more than 8 out of 10 cases.

"Every model we tested failed on the vast majority of cases," lead author Arya Rao, a Harvard medical student, told The Register. "That's the stage where uncertainty matters most, and it's where these systems are weakest."

Coauthor Dr. Marc Succi, a radiologist at Massachusetts General Hospital, went further: "Our results suggest today's off-the-shelf LLMs should not be trusted for patient-facing diagnostic reasoning without structured comprehensive human review." Succi added that models "can project confidence without showing robust reasoning," which he said can inflame anxiety in patients already worried about a symptom.

Rao offered a softer reading of her own data. Failure in the paper meant the model did not produce a fully correct differential, not that it was entirely wrong. Raw accuracy ranged from 63 to 78 percent, which she said suggests models were often partially correct. The team still argues the stricter metric matters, because these systems are being marketed as frontline agents that narrow down diagnoses before a human takes over.

Membership inference, near-perfect at the individual level

The second problem sits underneath the first. On 24 June 2026, German researchers published a Nature paper showing that discriminative medical AI models, the kind used to classify data and predict on new inputs, readily reveal whether a specific patient's record was part of their training set.

The technique is called a membership inference attack. It works because a model tends to be more confident on inputs it has seen before. An attacker feeds it patient data, watches the confidence, and infers inclusion.

Across seven medical datasets covering images, ECG records and electronic health records, the team found individual patients could be identified with "near-perfect attack success." The paper's lead author, Moritz Knolle of the Technical University of Munich, told The Register that patients from underrepresented groups, by race, insurance status, sex, imaging protocol or disease status, are easier to single out than those who do not stand out.

"Generally speaking, privacy risks from MIAs become more severe as a model's training cohort becomes more specific," Knolle said. He gave the example of a dataset whose membership itself reveals a dormant genetic condition such as Huntington's disease, depression, or attendance at a specialised treatment clinic.

"In our paper we show that an attacker with partial access can still successfully conduct MIAs."

Knolle also undercut the usual defence that anonymised records are safe. "An attacker conducting a MIA does not need to know who the data belongs to," he said. "In fact, all the datasets we use in our study were anonymized." Partial blood test results would be enough, he added, and given how often healthcare data leaks, obtaining them is not a stretch.

What the regulators are being asked to do

The Nature team wants two changes. First, privacy audits should measure risk at the individual patient level rather than in aggregate. The standard protocol, in the researchers' words, does not adequately capture what near-perfect per-patient attack success actually means. Second, they recommend differential privacy frameworks, which mathematically bound what a model can reveal about any single training record. Knolle said he hopes the medical AI community "will start to take privacy risks seriously."

Set that against the accuracy side. Microsoft's MAI Diagnostic Orchestrator, announced 30 June 2025, hit 85.5 percent on 304 complex New England Journal of Medicine cases against 21 physicians who scored 20 percent, according to GeekWire. WIRED, citing the same work, put the AI at 80 percent and reported a 20 percent cost reduction. Microsoft AI CEO Mustafa Suleyman called it "a big step towards medical superintelligence."

MIT scientist David Sontag told WIRED the paper was strong on methodology but warned the doctors were barred from using outside tools, which does not reflect real practice. Eric Topol of Scripps called the cost finding novel. Both said a clinical trial with real patients is the next step. Microsoft has not decided whether to commercialise the tool.

So the picture regulators face is not one failure but two, moving in opposite directions: systems that are weakest exactly where uncertainty is highest, and systems that are most revealing exactly where their training data is most specific.