(D) The Impact of Longitudinal Data-Completeness of Electronic Health Record (EHR) Data on Prediction Performance of Clinical Risk Scores

Publication Number: 662

Background: Electronic health record (EHR) discontinuity, i.e., receiving care outside of a given EHR system, has been shown to cause substantial misclassification in commonly used clinical risk scores. A prediction model was previously developed to identify patients with high EHR-continuity, but the impact of this algorithm on the prediction performance of these scores when predicting clinical outcomes is unknown.

Objectives: To assess the performance of commonly used clinical risk scores when predicting clinical endpoints in patients with high vs. low predicted EHR-continuity.

Methods: The study cohort was patients aged ≥ 65 years identified in a US multi-center EHR linked with Medicare claims data (2007/1/1 – 2017/12/31). Included subjects were required to have 365-day Medicare (Parts A, B, D) continuous enrollment, with at least one overlapping encounter in the study EHR. The index date was defined by the date at which these two criteria were met. Four risk scores were calculated based on covariates assessed in 365 days prior to the index date using EHR data alone (subject to misclassification due to EHR-discontinuity). We examined the performance of 1) combined co-morbidity score (CCS) predicting death 2) claim-based frailty score (CFI)predicting death 3) CHAD2DS2-VASc predicting ischemic stroke 4) HAS-BLED predicting bleeding using logistic regression. All the clinical endpoint (death, stroke, and bleeding risks) were evaluated in the linked EHR-claims data (not subject to misclassification due to EHR-discontinuity) in one year following the index date. We compared prediction performance of risk scores based on Area under the ROC curve (AUC), stratified by four levels of EHR-continuity score with cut-off informed by the literature (low: < 0.055, low-intermediate: 0.055-0.218, intermediate: 0.218-0.371, high: ≥0.371).

Results: The study cohort included 319,740 patients (42,066 atrial fibrillation patients for CHAD2DS2-VASc and HAS-BLED scores). Using CCS based on EHR alone to predict one-year risk of death, the AUC was 0.562 in the lowest EHR-continuity level and increased to 0.838 in the highest EHR-continuity level. The corresponding improvement in AUC was 0.527 to 0.699 for CFI predicting death, 0.605 to 0.757 for CHAD2DS2-VASc and ischemic stroke, and 0.549 to 0.750 for HAS-BLED and bleeding. In contrast, AUC remained stable in the models that used risk scores measured from linked EHR-claims data.

Conclusions: Using EHR data only, the prediction performance of risk scores is substantially worse in patients with low predicted EHR-continuity vs. those with high predicted EHR-continuity. Restricting the study cohort to those with high EHR-continuity yielded a similar risk score prediction performance when based on EHR alone vs. EHR-claims data.