(W1230-02-12) Computational Prediction of Drug-Induced Liver Injury with Combination of Convolutional Neural Networks and Multilayer Perceptron

Purpose: Drug-induced liver injury (DILI) has been one of the major concerns to pharmaceutical industries, clinical researchers, and regulatory scientists because it has contributed to drop clinical success rates of investigational drugs and onetime had been one of the main reasons leading to acute liver failure in the United States. For these reasons, a lot of efforts have been made to develop causality assessment method for DILI. As results, new tools like the Roussel Uclaf Causality Assessment Method (RUCAM) and Hy’s raw have been adopted broadly in clinical studies and regulatory affairs. However, these methods can be applicable only in clinical investigations. Most pharmaceuticals have expected some simplified methods to assess DILI potentials of their compounds in laboratory levels because of time and expense issues. For this purpose, computational tools could be valuable replacement to accelerate drug development and save the cost simultaneously.

Methods: For the development of computational DILI-prediction model, DILIrank dataset, LiverTox dataset, dataset from Xu et al. (Xu dataset), and dataset from Greene et al. (Greene dataset) were chosen as DILI data sources. After some preprocessing such as removal of mixtures, metallo-drugs, and redundant compounds, DILIrank dataset and LiverTox dataset were combined into training dataset and the remaining datasets were integrated into validation dataset (Table 1). In DILIrank dataset, drugs labeled as vMost-DILI-Concern or vLess-DILI-Concern were classified to DILI positives and those labeled as vNo-DILI-Concern to DILI negatives. In LiverTox dataset, drugs with a likelihood score of “A” or “B” were classified to DILI positives and those with “E” or “E*” to DILI negatives. Xu et al. had already classified their compounds as DILI-positives or DILI-negatives. Greene et al. categorized their compounds into four classes; only drugs labeled with “not evidence (NE)” and “human hepatotoxicity (HH)” were chosen and classified as DILI-negatives and DILI-positives, respectively. Overall, the training dataset consisted of 479 DILI-negatives and 461 DILI-positives, and the validation dataset was composed of 74 DILI-negatives and 105 DILI-positives. Convolutional neural networks (CNN) model and multilayer perceptron (MLP) model were developed based on 1D and 2D PaDEL descriptors. Following removal of completely zero-filled descriptors and data normalization to [-1, 1], 625 descriptors that were statistically significant by p-value from two-tailed t-test were selected. The architecture of CNN model and MLP model were shown in Figure 1. The models were trained with stratified 10-fold cross-validation and validation dataset were grouped into three subsets by extended connectivity fingerprint 4(ECFP4)-based similarities (10%, 30% 50%) with training dataset by DILI class and the performance was measured on accuracy, area under the curve (AUC), precision, recall, and f1 scores. At combination prediction, only when predicted labels for compounds from both models were DILI negatives, they were classified as DILI negatives and if not, they were classified as DILI positives.

Results: To overview structural similarities among datasets or between DILI classes, ECFP4 integer bits where each bit represents single atom or substructure were calculated for each compound and distribution of the bits were examined. Figure 2 showed that there were some substructures that exist in training dataset but not in validation dataset or vice versa. For this reason, we created three subsets from the validation dataset by applying the minimum structural similarity to 10%, 30% and 50% by DILI class. 173 compounds (71 DILI-/102 DILI+) were retrieved at 10% similarity, 102 compounds (35/67) at 30%, and 48 compounds (14/34) at 50%. For the CNN model, accuracies were 0.572, 0.735, 0.854 and AUCs were 0.535, 0.714, 0.887 for each validation subset. For the MLP model, accuracies were 0.572, 0.706, 0.813 and AUCs were 0.562, 0.732, 0.887 (Table 2, Figure 3). At combination prediction, accuracies were measured to 0.583, 0.726, 0.854, which were similar or slightly higher results. Especially remarkable results were observed in recall and f1 score. The recall of 0.618 in CNN model and 0.637 in MLP model at 10% similarity were increased to 0.706 in combination prediction. F1 score was also improved to 0.667 that was 0.63 in CNN model and 0.637 in DNN model. Such better performance results were also examined at 30% and 50% similarity subsets. The recall was enhanced from 0.761 and 0.746 to 0.821 and f1 score was from 0.791 and 0.770 to 0.797 at 30% similarity subset. The f1 score was increased from 0.882 and 0.853 to 0.912 and f1 score was from 0.896 and 0.866 to 0.899.

Conclusion: In this study, the two basic neural networks of CNN and MLP were tested to evaluate DILI potential of drugs. The two models showed better performance on higher structural similarity subsets. When the prediction results were combined, much higher recall and f1 score were achieved. These results implies that it would contribute to generate more robust model to define proper model coverage and to integrate individual models.

References: 1. Hay, M.; Thomas, D.W.; Craighead, J.L.; Economides, C.; Rosenthal, J. Clinical development success rates for investigational drugs. Nat. Biotechnol. 2014, 32, 40–51, doi:10.1038/nbt.2786.
2. Harrison, R.K. Phase II and phase III failures: 2013–2015. Nat. Rev. Drug Discov. 2016, 15, 817–818, doi:10.1038/nrd.2016.184.
3. Reuben, A.; Koch, D.G.; Lee, W.M.; Group, A.L.F.S. Drug-induced acute liver failure: results of a U.S. multicenter, prospective study. Hepatology 2010, 52, 2065–2076, doi:10.1002/hep.23937.
4. Fontana RJ, Watkins PB, Bonkovsky HL, et al. Drug-Induced Liver Injury Network (DILIN) prospective study: rationale, design and conduct. Drug Saf. 2009;32(1):55-68. doi:10.2165/00002018-200932010-00005
5. Temple R. Hy’s law: predicting serious hepatotoxicity. Pharmacoepidemiol Drug Saf. 2006;15(4):241-243. doi:10.1002/pds.1211

Acknowledgments:

Funding: This research was supported by the Korea Institute of Toxicology, Republic of Korea (1711159817).
Conflicts of Interest: The authors declare no conflict of interest.

Table 1. Datasets used to generate drug-induced liver injury (DILI)-prediction model in this study



Table 2. Model performance of CNN model, MLP model, and combination prediction



Figure 3. ROC curves for the validation subset at similarity of 10%, 30%, and 50%