E0173 - Validation of a Deep Machine Learning Tool to Determine Intra-Procedural Screening Colonoscopy Quality Indicators in an Academic Health System

Artin Galoosian, MD, MA¹, Jamie O. Yang, MD¹, Emma Peterson, MS², Cleo K. Maehara, MD, MMSC², Jayraan Badiee, MPH¹, Camille Soroudi, MD², Anthony Myint, MD², Yuna Kang, MD³, Bita V. Naini, MD⁴, Sadie De Silva, MD⁵, V. Raman Muthusamy, MD, MAS, FACG⁶, Eric Esrailian, MD, MPH¹, William Hsu, PhD⁷, Folasade P. May, MD, PhD, MPhil^8
1University of California, Los Angeles, Los Angeles, CA; ²UCLA, Los Angeles, CA; ³University of California Los Angeles, Los Angeles, CA; ⁴DGSOM, Los Angeles, CA; ⁵UHS SoCal MEC, Temecula, CA; ⁶University of California, Los Angeles, Sherman Oaks, CA; ⁷David Geffen School of Medicine at UCLA, Los Angeles, CA; ⁸UCLA, David Geffen School of Medicine, Los Angeles, CA

Introduction: High quality screening colonoscopy is the hallmark of effective colorectal cancer (CRC) prevention. However, continuously monitoring colonoscopy quality indicators for providers and health systems remains a challenge. We developed and validated a natural language processing (NLP) tool to automatically measure 4 intraprocedural colonoscopy quality improvement (QI) metrics and characterized its performance.

Methods: We implemented this quality initiative in a large academic healthcare system that performs >15,100 screening colonoscopies yearly in 6 endoscopy centers. We trained and developed an NLP algorithm that extracts and analyzes data from free-text colonoscopy reports to measure colonoscopy indication (IND), bowel preparation (BP), cecal intubation (CI), and successful cecal intubation (SCI) (Figure). We then randomly selected 600 screening colonoscopies performed between 6/2020-2/2021 to validate the NLP’s performance. We compared the NLP-derived quality metrics to manual chart review (gold standard). We calculated the sensitivity, specificity, positive predictive value, negative predictive value, F-score, and accuracy for each metric. When NLP and manual review were discordant, another physician repeated manual review to resolve the discrepancy.

Results: Our validation cohort (n=600) was 49.2% female with mean age 61.5 (sd=8.9, Table). Overall, the NLP had excellent performance across all four evaluated quality metrics when compared to manual chart review. For all metrics, sensitivity ranged from 99.3 to 100.0% and specificity ranged from 94.3 to 100.0% (Table). Within our cohort, the NLP misclassified only 2 cases for the documentation of IND. For documentation of BP, the NLP misclassified 1 case. Both misclassifications (IND and BP) were due to conflicting documentation by the endoscopist in the same colonoscopy report. The NLP had perfect performance for the documentation of CI. Finally, for SCI, NLP misclassified 12 cases, mainly due to the endoscopist not mentioning the word “cecum” or documenting “terminal ileum” instead.

Discussion: We developed an automated NLP algorithm that is highly accurate and sensitive in determining four priority intraprocedural colonoscopy quality indicators. Metrics from this tool can inform where to invest resources to further improve QI measures. In the future, we hope to optimize the NLP performance, measure additional colonoscopy quality metrics, and disseminate the NLP algorithm to other health systems to improve CRC outcomes.




Disclosures:


Artin Galoosian, MD, MA¹, Jamie O. Yang, MD¹, Emma Peterson, MS², Cleo K. Maehara, MD, MMSC², Jayraan Badiee, MPH¹, Camille Soroudi, MD², Anthony Myint, MD², Yuna Kang, MD³, Bita V. Naini, MD⁴, Sadie De Silva, MD⁵, V. Raman Muthusamy, MD, MAS, FACG⁶, Eric Esrailian, MD, MPH¹, William Hsu, PhD⁷, Folasade P. May, MD, PhD, MPhil⁸. E0173 - Validation of a Deep Machine Learning Tool to Determine Intra-Procedural Screening Colonoscopy Quality Indicators in an Academic Health System, ACG 2022 Annual Scientific Meeting Abstracts. Charlotte, NC: American College of Gastroenterology.