E0005 - Comparison of Machine and Deep Learning Methods With the Mayo Clinical Risk Score in Predicting 1-Year Mortality in Primary Biliary Cholangitis Patients

Tuesday, October 25, 2022

3:00 PM – 5:00 PM ET

Location: Crown Ballroom

Presenting Author(s)

Shivabalan Kathavarayan Ramu, MBBS

Mayo Clinic
Rochester, MN

Shivabalan Kathavarayan Ramu, MBBS¹, Anjali Byale, MBBS¹, Achintya Singh, MBBS²
¹Mayo Clinic, Rochester, MN; ²Cleveland Clinic, Rochester, MN

Introduction: To compare machine learning (ML) and deep learning (DL) approaches to predict survival outcome in patients at 1- year with Primary Biliary Cholangitis (PBC) and to compare with the Mayo Clinical risk score

Methods: The data is taken from a randomized trial in PBC patients conducted between 1974 and 1984 by the Mayo Clinic. The first 312 cases in the dataset were included out of a total of 424 PBC patients referred. The rest that contained subjects with >30% missing data, were excluded. We evaluated the following models: 1) The 5-covariate Mayo model: log(bilirubin), albumin, log(prothrombin time), edema, and age, 2) 17-covariate Cox proportional hazards model by componentwise likelihood based boosting with stepnumber 10 and penalty number 100, 3) 17-covariate Deephit: DeepHit: trains a neural network to learn the estimated joint distribution of survival time and event, while capturing the right-censored nature inherent in survival data. Analysis was done with frac 0.3, relu activation, 0.1 dropout, 100L epochs and a batch size of 32L, 4) 17-covariate Multitask logistic regression model with ranking based feature selection to predict survival using a logistic regression model and the parameters from each model are estimated simultaneously in the maximization of the joint likelihood function, 5) random survival forest with 1000 trees and 6) Support vector machine. For the ML & DL models all 17-covariates were include in the analysis. Analyses were done in RStudio, and missing values were imputed using missRanger package.¹ The data is split into 80 percent training and 20 precent validation. All models were 5-fold cross validated. Prediciton statistics was calculated for each model developed. Finally, the ML & DL models were compared with the 5-covariate model of the Mayo Clinic Risk score.

Results: Multitask logistic regression model showed the highest AUC (1.00) with a Harrell’s C-statistic of 0.90. Cox boost and random survival forest had equal AUCs (0.87) and C-indices (0.84). But the 5-covariate model of Mayo clinic Risk score had a higher AUC when compared to cox boost, random survival forest, support vector machine or even Deep hit.

Discussion: Machine learning methods offered limited improvement over the Mayo Clinical risk score except for the Multitask logistic regression model in predicting PBC survival outcome at 1-year.

S.No.	Model specification	AUC	95% CI	Harrell’s C-statistic
1.	5-covariate model of Mayo clinic Risk score	0.88	0.85-0.91	0.79
2.	Cox boost	0.87	0.84-0.90	0.84
3.	Deep hit	0.70	0.66-0.74	0.30
4.	Multitask logistic regression	1.00	1-1	0.90
5.	Random survival forest	0.87	0.84-0.90	0.84
6.	Support Vector Machine	0.56	0.52-0.61	0.43

Table: Table 1. Performance of Risk Scores from the 5-Covariate Model in comparison to the machine learning and deep learning models at 1-year

Disclosures:

Shivabalan Kathavarayan Ramu indicated no relevant financial relationships.

Anjali Byale indicated no relevant financial relationships.

Achintya Singh indicated no relevant financial relationships.

Shivabalan Kathavarayan Ramu, MBBS¹, Anjali Byale, MBBS¹, Achintya Singh, MBBS². E0005 - Comparison of Machine and Deep Learning Methods With the Mayo Clinical Risk Score in Predicting 1-Year Mortality in Primary Biliary Cholangitis Patients, ACG 2022 Annual Scientific Meeting Abstracts. Charlotte, NC: American College of Gastroenterology.