(I) Development of a Python Notebook to Automate Predictive Modeling: Total Knee Arthroplasty Case Study

Publication Number: 1008

Background: Predictive models (PMs) using machine-learning are increasingly applied to medical decision-making. Rapid analytics, comparison of several models, and identification of best-fitting models using automated tools can improve prediction.

Objectives: We developed a novel python notebook to automate modeling steps and allow the intuitive presentation of results. A total knee arthroplasty (TKA) dataset was used as a case study.

Methods: An analytical dataset (ADS) from the Medicare standard analytical files (SAF), comprising all patients with primary, elective total knee arthroplasty, from 2016 to 2019, was created. Two outcomes were analyzed: 90-day readmissions with knee-related complications (RKC) and 90-day readmissions for cardiovascular complications (RCC). These targets were selected as they represent very different types of risks and predictability based on pre-operative patient characteristics. Using a notebook developed in Python, the following steps were automated: 1) exclusion of variables with > 20% missingness; for all other missing data, medians were imputed for numerical and modes for categorical variables, 2) trimming tails at 99.998%, 3) splitting the ADS into training (70%) and test (30%) datasets, and 4) removal of highly correlated variables (VIF > 10). Six PMs were analyzed: Logistic regression classifier, decision tree, random forest, weighted XGBoost (WXGB), weighted CatBoost (WCB), and Support Vector. Ensemble modeling was performed with voting classifier. Results for each model included metrics of F1 scores, accuracy, balanced accuracy, precision, sensitivity, specificity, area under the curve (AUC), and Kappa. The best model fit was determined by high sensitivity and AUC.

Results: The ADS included 683,695 TKA patients with 108 pre-operative variables. All models except for the SVM could be generated. Outcome data was highly imbalanced (ie, low 90-day rate): 2% of patients had an RKC, and 13% of patients had an RCC. For RKC, WCB had the highest sensitivity (0.557) and AUC (0.625) while all models had weaker scores. The ensemble modeling had lower sensitivity and AUC than WCB. For RCC, WCB also outperformed all other models with an AUC of 0.837 and a sensitivity of 0.776. The comparatively high AUC predictability of RCC vs RKC was possibly due to the lower rate of RKC vs RCC. In addition, cardiac readmissions are more likely predicted by variables in the ADS (eg, preoperative comorbidities) vs knee readmissions, which might be associated with surgical characteristics not captured in a claims database.

Conclusions: With the increasing automation of predictive analytics, the development of tools that simultaneously provide top-line results and in-depth access to methods and decision steps is critical.