(153) A Pragmatic Comparison of Logistic Regression vs. Machine Learning Methods for Propensity Score Estimation

Publication Number: 1198

Background: In pharmacoepidemiologic research, it is a common practice to use logistic regression to estimate the propensity score (PS). However, logistic regression assumes linearity in the logit, and users typically do not examine this assumption. In contrast, machine learning algorithms (MLA) implicitly consider complex covariate structures and non-linear relationships within highly dimensional data and have been used increasingly in PS modeling.

Objectives: Using real-world data, we compared the performance of commonly used MLA to conventional logistic regression in the context of PS matching: random forest, gradient boosting machine (GBM), and extreme gradient boosting (XGBoost).

Methods: The study population included adult patients with rheumatoid arthritis (RA) who were newly initiated on baricitinib (n=255) or a tumor necrosis factor inhibitor (TNFi) (n=1,304) from 2018 to 2021 with ≥ 6 months continuous enrollment prior to treatment initiation in a large US administrative claims database. A total of 22 patient characteristics including demographics, comorbidities, medication use, and RA severity were assessed in all PS estimation approaches. We compared the performance of each PS estimation method by examining number of imbalanced covariates (standardized mean difference ≥0.1, evaluated via bal.tab function in R) after matching, and the extent of matching-related attrition in the treated group.

Results: There were six imbalanced covariates in the unmatched sample. After PS matching on a 1:3 ratio, both logistic regression and random forest achieved zero imbalanced covariates with similar treatment group attrition (~52%) regardless of caliper width used. GBM and XGBoost each yielded one imbalanced covariate with less attrition in this context, especially XGBoost (9% when 0.1 caliper was used and 32% when 0.01 caliper was used). In the context of 1:1 matching, logistic regression performed the best in terms of covariate balance (zero imbalanced covariates vs. one in each MLA), and XGBoost yielded the highest number of matched patients in the treatment group, with both calipers of width 0.1 (n=254, < 1% attrition) and 0.01 (n=227, 11% attrition).

Conclusions: Logistic regression and MLA for PS estimation both demonstrated adequate performance in this study, with small deviations in covariate balance. However, matching-related attrition was highly variable between PS estimation methods and this may have implications for study precision. Future analyses in different study populations and in simulation studies with various assumptions of non-linearities are warranted.