(160) Random Forests Quantile Classifier, a Novel Proposal to Account for Imbalanced Data Sets in The Prediction Model Development: A Feasibility Study Using Retrospective Registry Data of Kawasaki Disease in Japan
– Kyoto University Graduate School of Medicine and Public Health, Kyoto-Shi, Kyoto, Japan
Background: Machine learning (ML) is an atractive tool to develop a prediction model of the treatment response. Class imbalance is a known problem for ML-based prediction models, commonly favoring a higher accuracy for the majority class at the expense of the minority class.
Objectives: To test the feasibility of the recently proposed approach for imbalanced data sets: random forests quantile classifier (RFQ).
Methods: This study uses retrospective registry data of Kawasaki disease (KD) — the most common childhood vasculitis of unknown etiology — in Japan. This data set includes 599 responders (77.9%) and 170 non-responders (22.1%) to the standard treatment of intravenous immunoglobulin (IVIG) infusion. Non-response indicates a high risk of cardiovascular sequelae due to KD, and patients predicted to be non-responders are candidates for intensified therapy during the early-stage disease course. Two random forest (RF)-based models were constructed to predict IVIG unresponsiveness: traditional RF (i.e., without addressing class imbalance) and the RFQ-based method. For these models, the misclassification rates were calculated and compared. The randomForestSRC R package (ver. 3.0.1) was used for the computation.
Results: In the traditional RF model, the misclassification rate was 21.3% overall: 1.5% in responders and 91.2% in non-responders. In contrast, the RFQ model The traditional RF model resulted in a misclassification rate of 21.3% overall: 1.5% in responders and 91.2% in non-responders. In contrast, the RFQ model outputted an overall misclassification rate of 33.3%: 32.1% in responders and 37.7% in non-responders.
Conclusions: For this imbalanced data set, the traditional RF model achieved a high accuracy for the responder class at the cost of poor predictive performance for the non-responder class. In contrast, the RFQ-based model obtained a similar misclassification rate for the responders and non-responders, although the overall misclassification rate was higher. This finding demonstrates the utility of RFQ for situations entailing class imbalance, particularly when a high misclassification rate in the minority class is detrimental, as in the case of the prediction model for the KD treatment response.
