O2006: Performing Rapid-cycle Randomized Controlled Trials to Evaluate Ehr-based Quality Improvement Interventions

Purpose/Objectives: Rapid-cycle randomized controlled trials offer a rigorous method for evaluating electronic health record (EHR) quality improvement interventions. In this study we demonstrate one such method to perform practical randomized controlled trials of clinical decision support tools by stratifying on provider ID. As an example, we present preliminary results from an experiment evaluating the effectiveness of an interruptive alert aimed at reducing the use of free-text in discharge medication prescriptions. Free-text entries in discrete prescription fields can lead to medication errors and inequity by bypassing EHR safety checks, impairing translation of medication labels for non-English speaking patients, and resulting in missing information (e.g. dose, route, frequency).

Design/Methods: This study occurred at a standalone academic children’s hospital employing the Epic EHR (Epic Systems, Verona, WI) between December 15, 2021 and April 15, 2022. Two versions of an EHR alert—one “active” and one “silent”—were created that triggered when a provider attempted to sign a discharge medication prescription with free-text in a discrete field. The active version displayed an interruptive alert to the user (Figure 1), and the silent version triggered in the background, serving as a control. Providers were assigned to the active and silent arms based on EHR provider IDs (even IDs received the active alert). The proportion of final prescriptions with free-text was calculated for each group and compared. Statistical significance was tested using a two-tailed chi-squared test of independence. Weekly triggering rates were measured and plotted in a run chart. A Mann-Kendall test was used to determine if there was a statistically significant trend. All p-value thresholds were 0.05.

Results: Over the 16-week study period, 221 active and 240 silent alerts were triggered. In the experimental group, 186/221 signed, final prescriptions included free-text (84%), whereas in the control group, 222/240 signed, final prescriptions included free-text (93%), a statistically significant difference (p = 0.01). There was no significant trend in alert rates over time in the active (p = 0.80) or silent groups (p = 0.74) (Figure 2).

Conclusion/Discussion: This study demonstrates a practical method for performing rapid-cycle randomized controlled trials in evaluating EHR-based quality improvement interventions. In this example, an interruptive alert reduced the occurrence of signed free-text prescriptions. In the control group, 93% of attempted free-text prescriptions included free-text in their final form. This is likely due to other clinical decision support tools that may have triggered a change to the final prescription. In the experimental group, this rate was 9% lower, which, owing to the randomized design of the study, is attributable to the interruptive alert. The alert triggering rate, however, did not meaningfully decrease in the experimental group over time, suggesting the educational component of the alert did not have a lasting effect on prescribing behaviors.