Pathologists must examine certain assumptions behind artificial intelligence (AI) systems—particularly the belief that better performance requires ever-larger data sets—and consider how data collection and annotation shape the clinical insights these tools provide, said Takunda Matose, MBe, PhD. In a plenary lecture Friday morning at the Association for Molecular Pathology’s annual meeting in Boston, Dr. Matose emphasized that creating AI systems that improve patient care will require deliberate and thoughtful oversight from those who develop and use them.
Dr. Matose—a research ethicist at Cincinnati Children’s Hospital and Medical Center, assistant professor of pediatrics at the University of Cincinnati Medical Center, and philosophy instructor at the University of Cincinnati—stressed in his lecture “Ethical Frontiers: The Promise and Perils of Healthcare AI in a Socially Connected World” that both clinicians and developers must think differently about the relationship between data volume, algorithmic accuracy, and responsible use (Figure 1).
“What I’m going to try to convince those of you who are not thinking about ethics all the time or most of the time is that maybe you should think about ethics a lot more often,” he said. “From a purely pragmatic standpoint, I think it’s going to improve technological performance. For those of you who are developers, it’s going to improve the development side. For those of you who are pathologists, it’s going to help with the communication of the information that you’re getting.”
The Challenges of AI
Dr. Matose challenged the widespread assumption that adding more genomic, imaging, and clinical data automatically produces better AI models. He shared estimates showing that health care now generates between 14% and 30% of all global data (Figure 2)—yet only a small fraction is analyzed or used meaningfully.
“We’re told we need more of it—we need more and more and more data,” he said. However, he noted that roughly 90% of health care data goes unused, and much of what is captured was not designed with model development in mind.
He proposed that AI should be understood as a tool for performing complex conditional-probability calculations—calculations that shift with each new input, pipeline choice, annotation step, or contextual variable. To illustrate this, he walked the audience through the classic Monty Hall probability puzzle (Figure 3), noting that AI systems can remove options known to be wrong—“take away bad options”—but cannot guarantee a correct answer. “There’s no guarantee of success either way,” he said. “You could still get the prize if you stick with your original choice. Alternately, if you switch, there’s no guarantee of getting the prize.”
Bias is inescapable when it comes to AI systems, he said, because it’s built into the parameters of operation for these systems. “All the decisions [made when] designing the systems—which data sets we chose to select, what our training sets are going to be, what our validation sets are, the way we’re annotating the data—these all introduce limitations in terms of the kinds of outputs that we're going to get.” And even if a system could be designed with no bias, there is still a possibility of introducing biases when it is used, he observed. The problem with biases is that they can lead to errors due to mistaken reasoning (Figure 4).
Another issue is a tendency to overinterpret accuracy metrics provided by vendors or researchers. “We talk about accuracy as though it’s a singular thing,” he said. But the usefulness of any metric depends on its context, the population it was validated in, and the specific task the system is performing, Dr. Matose noted. Even strong validation data, he said, should not be mistaken for guarantees. “Past performance is only probabilistically predictive of future performance.”
AI’s probabilistic capabilities can greatly enhance clinical workflows when applied appropriately, because these tools excel at detecting subtle patterns across large data sets, generating probabilistic inferences, and supporting decision-making in ways humans cannot consistently achieve. Dr. Matose cautioned, however, against reflexively placing humans “back in the loop,” because human judgment carries its own biases and inconsistencies. Instead, human discretion must be applied thoughtfully to maximize the strengths of both humans and AI.
The Ethical Frontier
Dr. Matose encouraged attendees to adopt what he called an “ethical lens” when it comes to AI, with a focus on collective benefits and burdens, institutional policies, and obligations to patients, families, health care providers, and other stakeholders. Due to the probabilistic nature of AI, some patients will inevitably fall outside the model’s predictive success, he cautioned.
“There’s always going to be someone for whom these things are not going to work. Everything has a failure rate,” and practitioners must take those who do not benefit into account.
Managing health care data responsibly is a central challenge. Dr. Matose noted that “data are only useful in aggregate” and must be treated as "social facts"—collected by humans, shaped by society, and meaningful only within social systems. They are not objective, unbiased truths. Genetic or clinical information about one person can reveal insights about families, communities, or populations, making privacy and security critical concerns. He highlighted the need for updated regulatory and conceptual frameworks that match today’s technological landscape, suggesting that institutions reconsider long-held assumptions about data collection, retention, and disclosure (Figure 5).
Another ethical dimension arises from the sheer number of stakeholders involved in health care AI tools. Dr. Matose contrasted vertical integration, which centralizes exposure but limits points of failure, with interoperability, which distributes control but also responsibility. Each approach carries trade-offs, he said, and institutions must weigh these carefully. He also underscored the importance of principles like FAIR (Findable, Accessible, Interoperable, Reusable), noting that operationalizing such standards becomes increasingly complex as technology evolves.
AI must be understood as a tool for probabilistic reasoning, which means it will never be 100% accurate, said Dr. Matose. These limitations are a natural consequence of working with probabilities. “We need to be upfront about what the limitations are, but there may be really good uses for these tools,” he said. “I'm not anti-AI. I use AI all the time, I think we need more of it, and it's going to be very helpful for reaching out to communities that don't have all the health care capacities that we might have at someplace like Cincinnati Children’s Hospital, a large academic medical center.”
Dr. Matose suggested that thoughtful judgment and decision-making may be more useful than simply placing humans “in the loop” of AI processes. Applying an ethical lens can help determine when smaller data sets are sufficient and remind users that health care is fundamentally social. “This will help us get the most out of AI without overpromising and underdelivering. And ultimately this will, I think, lead to better clinical experiences and outcomes,” he concluded.
