Fracture Risk Estimation in Post-menopausal Women: A Comparison of Machine Learning Methods and Software Systems

Background: Predictive modeling of health outcomes is a key tool for identifying high risk patients who may benefit from more intensive therapy or targeted quality improvement interventions.

Objectives: Large administrative claims databases are widely available and provide a wealth of data that can be used to train machine learning models; however, researchers building models from such data face many methodological and operational challenges. Using a large commercial claims database, we compared various machine learning methods and censored data approaches for estimation of 1-year fracture risk among women over the age of 50 years old. We found that commonly used machine learning methods were able to predict fracture risk on a large claims database. It is not clear how well these findings generalize to other data and research questions. Methodological challenges due to censored data in claims databases remain to be addressed.

Methods: Over 2.1 million women aged 50 years or older who had an outpatient medical encounter between January 1st, 2013, and September 30th, 2014, and who had 2 prior years of insurance coverage, were identified using an administrative health claims database consisting of members of large commercial and Medicare Advantage plans. Closed fractures for hip, spine, composite, and broad composite fracture types were identified during 1 year of follow-up, and covariates were identified during a 2-year baseline period. Prediction was performed using penalized logistic regression (Lasso), gradient boosted machine, distributed random forest, deep learning, and a stacked ensemble, both with and without inverse-probability of censoring weighting (IPCW) to account for loss of health plan enrollment. Training was performed on 80% of the data using 10-fold cross-validation, with the remaining 20% hold-out set used to assess model performance.

Results: Our models achieved c-statistics of 0.82, 0.76, 0.71, and 0.73 for the hip, spine, composite, and broad composite fracture endpoints, respectively. The Lasso method performed well across the various scenarios that we considered and was easily implemented. Use of IPCW did not have a substantial impact on the prediction results, although the impact may be difficult to measure. Calibration of the predictions tended to be good with some observed miscalibration at the highest risk levels.

Conclusions: We found that commonly used machine learning methods were able to predict fracture risk on a large claims database. It is not clear how well these findings generalize to other data and research questions. Methodological challenges due to censored data in claims databases remain to be addressed.