An artificial intelligence system generated highly accurate preliminary interpretations of urine drug tests and reduced laboratory sign-out time when integrated into routine clinical practice.
Researchers developed and evaluated the workflow using 83,553 urine drug tests from 26,459 patients performed at the University of Washington Medical Center between January 2014 and February 2024. Large language models (LLM) were used to extract structured substance-use labels from historical urine drug test interpretations, which were then used to train nongenerative machine learning models to predict substance use from immunoassay and mass spectrometry results. The predicted substance-use patterns were converted into preliminary clinical sign-out statements for review and editing by clinical chemistry fellows and attending physicians before final sign-out.
The primary outcomes were the accuracy of LLM–based label extraction, substance-use prediction, and AI-generated clinical interpretations, as well as the effect of the AI tool on workflow efficiency. Secondary outcomes included clinician modifications to AI-generated interpretations, postdeployment adoption, and model performance across demographic subgroups.
The LLM correctly extracted 99.9% of substance-use labels from historical clinical interpretations, outperforming manual labeling. The machine learning models predicted substance use with an area under the receiver operating characteristic curve greater than 0.99 for 23 of 26 substances, and prediction accuracy exceeded 94% for every substance evaluated. Model performance remained consistent across major demographic subgroups, with no subgroup area under the receiver operating characteristic curve below 0.97.
When integrated into a simulated clinical workflow, the AI tool reduced average sign-out time by 28.5 seconds per case, representing a 23% efficiency gain compared with standard workflow. When combined with an automated medication-list import feature, average sign-out time decreased by 65 seconds per case, corresponding to a 51% efficiency gain.
Workflow efficiency was achieved without reducting in interpretive accuracy. Among 198 AI-generated preliminary interpretations reviewed by laboratory readers, 70% required no substantive changes and 16% required only minor grammatical edits. Incorrect substance-use classification prompted overrides in 4% of cases, most commonly because the AI system did not have access to laboratory information below the mass spectrometry reporting threshold or because of complex cases involving sample adulteration. Following deployment, three of four laboratory readers used the AI interpreter for more than two-thirds of eligible cases, although adoption varied among readers and attending physicians.
The investigators also observed that prediction accuracy declined as the number of detected substances increased, reflecting greater case complexity. They emphasized that the system was designed to function within a human-in-the-loop workflow in which physicians retained responsibility for reviewing and approving every preliminary interpretation.
The study was limited by its retrospective, single-center design, lack of protocol preregistration, and relatively small postdeployment validation samples. The authors noted that external validation at institutions using different laboratory testing protocols and assays will be necessary before broader implementation.
Overall, the findings suggest that AI-assisted preliminary interpretation of urine drug tests may improve laboratory efficiency while maintaining high interpretive accuracy when incorporated into expert-reviewed clinical workflows.
"We found that AI-based interpretations were highly accurate, and when integrated into an existing workflow, led to substantial reductions in average sign-out time," wrote lead study author Nathan Laha, BSc, of the Department of Laboratory Medicine & Pathology, University of Washington School of Medicine, Seattle, and colleagues.
Disclosures: Andrew N. Hoofnagle, MD, PhD, reported receiving grants from Waters Inc, a mass spectrometry company, outside the submitted work. No other disclosures were reported.
Source: JAMA Network Open
