Medical AI privacy: when average protection is no protection at all
Moritz Knolle, a doctoral student at the Technical University of Munich, asked himself a question that led to an unsettling discovery: “I asked myself: would I contribute my data to train a medical AI model?”
To answer it, he and his colleagues ran a systematic test. They attacked seven real-world medical datasets, including 21,799 electrocardiograms and 377,110 chest radiographs, with membership inference attacks (MIAs), a technique that determines whether a specific person’s data was used to train a given AI model. The assumption in the field has been that MIA risk is low enough that, averaged across all patients, it presents no real threat.
That assumption, the team reports in Nature (DOI: 10.1038/s41586-026-10688-0), is dangerously misleading. The risk is not evenly distributed, and for some patients it is extreme.
How a membership inference attack works
Medical AI models are typically trained on large datasets of patient records, images, or physiological signals. For the model to perform well, it must learn patterns in the data. But learning patterns also means adjusting its internal parameters in ways that make it slightly more confident about the specific data points it has seen.
An MIA exploits this. The attacker, who has access to the trained model, feeds it a data point and records the model’s confidence score. If the confidence is significantly higher than what an equivalent model would produce without that data point, the attacker can infer that the data point was in the training set.
For medical data, that inference is dangerous. Knowing that a specific person’s data was used to train a model for, say, a rare cancer treatment tells an insurer or employer something that person may want kept private.
The distribution of risk
Previous studies reported MIA risk as an average across all patients, typically a low single-digit number. A model with 1% average risk seems safe. But Knolle and his collaborators, including Daniel Ruckert at TUM and Georg Kaissis at the Hasso Plattner Institute, found that the average hides a deeply unequal distribution.
Patients from minority groups underrepresented in the training data are heavily overrepresented in the 99th percentile of privacy risk. For these individuals, the MIA success rate can be dramatically higher than the average, in some cases approaching certainty for the most unusual patients.
The reason is statistical. A model learns the common patterns well, so the confidence it assigns to typical patients is similar whether or not they were in the training set. But for a rare patient, say someone with an unusual combination of conditions, the model’s confidence is a much stronger signal of membership. The model has seen so few examples like that patient that if it was in the training data, its influence on the model’s parameters is disproportionately large.
A paradox of scale
The problem is amplified by the very trend that makes medical AI useful. As models grow larger and more sophisticated, they analyze data more deeply, creating smaller and more detectable changes in predictive probability. Bigger models, paradoxically, may be less private.
Similarly, the more specific a training cohort, patients receiving a particular cancer treatment, for example, the larger the privacy risk for each member. The very specificity that makes a dataset valuable for answering a clinical question also makes its members more identifiable.
Differential privacy as a fix
The solution exists. Differential privacy (DP) adds a small amount of calibrated random noise during training, mathematically bounding how much information the model can leak about any individual. In the team’s tests, DP reduced MIA effectiveness while maintaining diagnostic accuracy.
But DP has not been widely adopted. Most hospitals and clinics that develop AI models do not implement it, either because they are unaware of the risk or because they consider the implementation effort unnecessary.
“The goal is to provide individuals with a statistical guarantee that no harm would come to you by being part of a dataset,” Knolle told Physics World. He and his colleagues are now expanding the work to generative AI models in healthcare, where the privacy risks may be even greater.
The core lesson of the study is mathematical, not technical. In privacy, as in medicine, averages can mislead. A risk that is low for the average patient provides no comfort to the patient for whom it is high, and until medical AI teams assess individual-level risk before deploying models, the patients with the most to gain from AI may also have the most to lose.
Reference: Knolle et al., Nature (2026). DOI: 10.1038/s41586-026-10688-0. Reported by Physics World, July 21, 2026.