Fewer than two percent of US Food and Drug Administration-cleared artificial intelligence medical devices reported randomized clinical trial data, according to a recent study.
Researchers conducted a cross-sectional analysis of all artificial intelligence and machine learning–enabled medical devices cleared by the US Food and Drug Administration between September 1995 and July 2023. Published in JAMA Health Forum, the researchers examined both premarket and postmarket benefit-risk reporting, including study design, efficacy, safety, bias assessments, adverse events, and recalls.
Using US Food and Drug Administration (FDA) decision summaries, device approvals databases, and the Manufacturer and User Facility Device Experience and Medical Device Recalls databases, the investigators identified 691 FDA-cleared artificial intelligence/machine learning devices. Most were classified as class II (moderate risk) and cleared via the 510(k) pathway (668 devices, 96.7%). Radiology accounted for most devices (n=531, 76.9%), followed by cardiovascular medicine (n=70, 10.1%) and neurology (n=20, 2.9%).
Premarket reporting was frequently incomplete. Study design was not described for 323 devices (46.7%), and sample size was missing for 368 (53.3%). Only 6 devices (1.6%) cited randomized clinical trial data, while 53 (7.7%) included prospective studies. Clinical performance outcomes were rarely reported: sensitivity was provided for 166 devices (24.0%), specificity for 152 (22.0%), and patient outcomes for just 3 (<1%). Demographic characteristics were disclosed in 31 device summaries (4.5%), and only 60 devices (8.7%) conducted a bias assessment.
Safety assessments were documented for 195 devices (28.2%), and adherence to international safety standards was noted in 344 (49.8%). Potential risks to health were reported in 42 devices (6.1%), primarily algorithmic errors (29, 4.2%) and user errors (30, 4.3%). Postmarket surveillance revealed 489 adverse events associated with 36 devices (5.2%), including 458 malfunctions, 30 injuries, and 1 death. In addition, 40 devices (5.8%) were recalled 113 times, most commonly for software issues (85 recalls, 75.2%). Devices with reported adverse events were more likely to be recalled than those without (38.9% vs 4.0%).
“These findings indicate that standardized efficacy, safety, and risk assessment reporting remains inadequate for FDA-cleared AI/ML devices, underscoring the need for dedicated regulatory pathways and robust postmarket surveillance to ensure patient safety,” noted lead researcher John C. Lin, BS, of the Department of Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, and colleagues.
Ravi B. Parikh, MD, MPP, reported research funding from federal agencies, foundations, and industry; personal fees, equity, and honoraria from multiple health care organizations; and unpaid board and editorial roles, all outside the submitted work, with no other disclosures reported.
Source: JAMA Health Forum
