O2010: The Deterioration Risk Index: Development and Prospective Validation of a Machine Learning Algorithm Predicting Pediatric Inpatient Deterioration

Background: Code blue events outside the ICU and emergency floor-to-ICU transfers have well recognized negative outcomes, often with identifiable precursors. Existing pediatric early warning systems often fail to identify children at risk for these deterioration events. By leveraging a multi-disciplinary team of data scientists, physician informaticists, EHR analysts, and clinicians, we developed and operationalized the Deterioration Risk Index (DRI), a machine learning algorithm predicting inpatient deterioration.

Methods: Model Development
Model development was conducted as a retrospective cohort study examining non-ICU pediatric inpatients from October 2015 to December 2019. Our prediction target was preventable clinical deterioration events, defined as code blue events outside the ICU, emergency floor-to-ICU transfers, and unexpected deaths. We developed individualized models for cardiac, malignancy, and general cohorts via repeated, cross-validated lasso-regularized logistic regression. Candidate model predictors included vital signs, supplemental oxygen, nursing assessments, diagnoses, lab results, and situational awareness program (Watchstander) criteria. Model performance was compared to PEWS and Watchstander using a novel simulation approach. Primary retrospective evaluation metrics included model sensitivity with at least two hours lead time and the number of alarms per detected event.

Implementation & Prospective Validation
DRI was piloted on the cardiology and general pediatric units in early 2021 followed by institutional go-live in August. DRI was incorporated into our mature Watchstander program to support the response arm of the alert. Prior to go-live, thorough workflow analyses, simulation sessions, and multi-disciplinary team engagements were completed with pilot units and units with anticipated high alert volumes. Performance was monitored regarding event identification, reduction, and operational impact compared to pre-implementation rates.

Results: In retrospective simulations, DRI was 2.4-times as sensitive as Watchstander (Figure 1) and required 2.3-times fewer alarms per detected event. The largest sensitivity gains were 4-fold and 3-fold among the cardiac and malignancy diagnostic cohorts when compared to Watchstander, respectively.

Pilot units have demonstrated a 73% decrease in preventable clinical deterioration events (p=.004). However, since hospital-wide go-live, we observed a non-significant increase in preventable deterioration events outside of pilot units (22% increase; p=0.816).

Conclusion: We developed and implemented a predictive algorithm to enhance identification of hospitalized children at-risk of deterioration outside of the ICU. Retrospectively, DRI outperformed our existing PEWS and Watchstander programs in providing advanced detection of those at risk. These results translated ‘from computer to bedside’ when considering algorithm performance and event reduction amongst pilot units. Unique considerations when interpreting prospective performance include: COVID-19, pandemic-related staffing shortages, and absence of typical pediatric illness seasonality.

Critical to success was acknowledgement of the unique pathophysiology of our diagnostic cohorts and engagement of multi-disciplinary teams to envision workflow integration and obtain buy-in. Our stepwise implementation approach also allowed EHR tool iterations and specialty-specific workflow modifications needed to maintain efficacy while minimizing operational burden.