(148) External Validation of a Machine Learning Algorithm to Distinguish Death from Disenrollment in Claims Data

Publication Number: 1193

Background: A common threat to validity in claims-based research is the inability to distinguish deaths from disenrollment events. An algorithm was developed using a national administrative claims database with linked date of death to predict death within 61 days of disenrollment (ROC = 0.933 in beneficiaries < 65 years).

Objectives: To conduct an external validation of an algorithm to predict death at disenrollment in order to understand performance of the algorithm in different study populations.

Methods: Validation data came from a large private health insurance provider in North Carolina (NC) from 2006-2018. Claims were linked to death records from the NC Department of Health and Human Services using a hierarchical matching algorithm. We identified a sub-population of beneficiaries aged 18-64 years with a pain diagnosis, surgery, or opioid analgesic and one year of continuous enrollment prior to disenrollment. Individuals were excluded if still enrolled at the end of the study period or 64th year of age. We evaluated model performance using C-statistics, sensitivity, and specificity overall and in subgroups of interest (age, sex, calendar year).

Results: We identified 1,224,073 individuals who disenrolled during the study period, of whom 14,190 (1.2%) died within 61 days of disenrollment. Overall model performance in the validation data was high (ROC: 0.899). At a predicted probability threshold of 80%, the algorithm classified 1.3% of disenrollments as deaths with sensitivity of 0.448 and specificity of 0.992. Sensitivity improved at lower probability thresholds. In subgroup analyses, model performance was a little higher among females (sensitivity: 0.490, specificity: 0.995, ROC: 0.926) than males (sensitivity: 0.422, specificity: 0.990, ROC: 0.870) at the 80% prediction probability threshold. Performance was highest among ages 55-64 (sensitivity: 0.591, specificity: 0.973, ROC: 0.890) and lowest among ages 18-34 (sensitivity: 0.042, specificity: 1.00, ROC: 0.815). Performance was comparable across calendar time.

Conclusions: Our external validation demonstrated that the algorithm performed similarly to performance in the original dataset used for algorithm development, with high specificity and varying sensitivity. This work indicates that this algorithm is generalizable to other commercial claims databases and populations of interest.