The Impact of Electronic Health Record Discontinuity on Prediction Modeling

Background: When electronic health records (EHRs) are used as data sources for developing prediction models, EHR discontinuity can cause information bias. An algorithm was previously developed to identify patients with high EHR continuity; use of this algorithm may improve the validity of EHR-based prediction modeling.

Objectives: To determine the impact of EHR discontinuity on the performance of prediction models.

Methods: Using longitudinal claims data from the Medicare system linked to EHR data from two medical care delivery networks, we set up system 1 as the training and system 2 as the validation data. The datasets consisted of patients aged ≥ 65 years with at least 365 days of continuous enrollment in fee-for-service Medicare programs and at least one EHR encounter overlapping with the Medicare enrollment period from 2007 to 2017. Based on predictors assessed by EHR alone, we built models predicting one-year risk of mortality, major cardiovascular (CV) events, and major bleeding events in patients with history of CV comorbidities, stratified by high vs. low algorithm-predicted EHR continuity. We used outcomes determined by EHR with claims data as the gold standard classification, as they are not subject to misclassification due to EHR discontinuity. The best performing models for each outcome were chosen among logistic regression, LASSO Regression, Random Forest, Gradient Boosting, and Bayesian Additive Regression Tree. We compared model performance by 5-fold cross-validated AUC and assessed optimism as the absolute difference between the AUC evaluated in the training vs. validation set.

Results: There were 26,342 patients in the high-continuity cohort (mean age 74.1 years, 48.9% female) and 25,935 patients in the low-continuity cohort (mean age 75.9 years, 46.5% female). When developing the prediction models, optimism was consistently greater in the low EHR continuity than in the high EHR continuity cohorts (optimism 0.0409 vs. 0.0264, 0.1707 vs. 0.1456, and 0.0290 vs. 0.0023 for mortality, CV events, and bleeding outcomes, respectively). In the validation set, the best performing model developed among high EHR continuity patients had a consistently higher AUC than that based on low continuity patients: AUC was 0.8101 vs. 0.7675 when predicting mortality (p < 0.001); AUC was 0.6380 vs. 0.5961 when predicting the CV events (p= 0.0012); AUC was 0.7255 vs. 0.6190 when predicting major bleeding (p=0.0705).

Conclusions: Among patients with prior CV comorbidities, when predicting mortality, major CV events, and bleeding outcomes, the prediction models developed in datasets with low EHR continuity consistently had worse performance compared to models developed with high EHR continuity.