P0266 - Validation of a Machine Learning Algorithm to Measure Adenoma Detection Rate From Screening Colonoscopies in a Large Health System

Artin Galoosian, MD, MA¹, Camille Soroudi, MD¹, Shailavi Jain, MD¹, Sartajdeep Kahlon, MD¹, Megan R. McLeod, MD, MS², Alex N. Kokaly, MD, MHSA¹, Jamie Yang, ¹, Emily Peterson, MS¹, Cleo Maehara, MD¹, Liu Yang, MD, MPH¹, Venkatara Muthusamy, MD, MAS, FACG¹, Eric Esrailian, MD, MPH¹, William Hsu, PhD¹, Folasade May, MD, PhD^3
1David Geffen School of Medicine at UCLA, Los Angeles, CA; ²David Geffen School of Medicine, UCLA, Los Angeles, CA; ³University of California Los Angeles David Geffen School of Medicine, Los Angeles, CA

Introduction: Adenoma detection rate (ADR) is inversely associated with risk of interval colorectal cancer (CRC), advanced-stage CRC, and CRC-related mortality. ADR is a critical screening colonoscopy quality indicator; however, it requires manual data extraction from colonoscopy and pathology reports which is labor-intensive. We developed a natural language processing (NLP) algorithm that automates ADR measurement at health system and provider levels to address this challenge. We then aimed to validate the NLP against manual chart review.

Methods: The study setting is a large urban academic health system that performs >15,100 screening colonoscopies annually in 5 endoscopy centers. We previously developed an NLP algorithm that uses machine learning techniques to extract free text from colonoscopy and pathology reports to measure colonoscopy quality indicators. We randomly selected 495 average-risk screening colonoscopy cases performed between 6/2020-2/2021 and conducted a manual chart review to collect patient characteristics and polyp features (location, size, histology). We then used manual chart abstraction and NLP algorithm approaches to determine ADR. We calculated sensitivity, specificity, positive predictive value (PPV), precision/recall via F-score, and accuracy of the NLP vs manual chart review as gold-standard.

Results: Our validation cohort (n=495) was 49.5% female (mean age 58.0, Table 1A). The overall health system ADR was 33.5% by NLP and 36.6% by manual chart review (p< 0.001). Although ADRs were statistically different between NLP and manual chart review in males (NLP 40.3%, manual 44.4%; p=0.046) and females (NLP 26.9%, manual 28.6%; p=0.004), the absolute difference was minimal (δ 1.7%-4.3%). Moreover, the NLP and manual chart review had an almost perfect agreement with overall Kappa coefficient of 0.93 (Table 1B). The ADR range for individual endoscopists (>20 colonoscopies) was 26.7%-43.5% by NLP and 26.7%-47.6% manually. The NLP algorithm was 91.2% sensitive, 99.7% specific, and had a PPV of 99.4%, F-score of 95.1%, and accuracy of 96.6% (Table 1B).

Discussion: ADR is a clinically important quality indicator but is challenging to measure. We developed and validated an NLP algorithm that automatically calculates screening ADR from colonoscopy and pathology reports. Our NLP algorithm is accurate, specific, and can easily be adapted and scaled for other health systems. It can also be used to help reduce human labor required for health systems to continuously monitor quality metrics.



Disclosures:


Artin Galoosian, MD, MA¹, Camille Soroudi, MD¹, Shailavi Jain, MD¹, Sartajdeep Kahlon, MD¹, Megan R. McLeod, MD, MS², Alex N. Kokaly, MD, MHSA¹, Jamie Yang, ¹, Emily Peterson, MS¹, Cleo Maehara, MD¹, Liu Yang, MD, MPH¹, Venkatara Muthusamy, MD, MAS, FACG¹, Eric Esrailian, MD, MPH¹, William Hsu, PhD¹, Folasade May, MD, PhD³. P0266 - Validation of a Machine Learning Algorithm to Measure Adenoma Detection Rate From Screening Colonoscopies in a Large Health System, ACG 2021 Annual Scientific Meeting Abstracts. Las Vegas, Nevada: American College of Gastroenterology.