Development of a Multivariate Prediction Model to Identify Possible Signal Triggering Reports from Spontaneous Individual Case Safety Reports

Background: The number of Individual Case Safety Reports (ICSRs) are rapidly increasing world-wide. The majority of ICSRs is reviewed manually to identify signal triggering reports (PSTR).

Objectives: To develop a prediction model to identify PSTR ICSRs before manual review. Secondly, to identify most important features to optimize data collection.

Methods: All ICSRs (n=30,424) received by the Netherlands Pharmacovigilance Centre Lareb between October 1st 2017 and February 26th 2021 were included. ICSRs originating from marketing authorisation holders or reported on vaccines were excluded, as these were processed differently. The outcome was defined as PSTR (yes/no), where PSTR ‘yes' was discussed at a signal detection meeting. 19 features were included, concerning information on: patients, adverse drug reactions (ADR) or drugs. Data was divided into a training (70%) and test set (30%) using a stratified split to maintain the PSTR/no PSTR ratio. Logistic regression, elastic net regression and eXtreme Gradient Boosting were trained and tuned on a training set. Random down-sampling was applied on the training set to adjust for the imbalanced dataset. Final models were tested on the test set. Model performances were assessed using the area under the curve (AUC) with 95% confidence interval of a receiver operating characteristic (ROC), and specificity and precision were assessed at a threshold for perfect sensitivity (100%, to not miss any PSTRs). Feature importance plots were inspected and a selection of features was used to re-train and test model performances with fewer features.

Results: 1439 (4.7%) of reports were PSTR. All three models performed equally with a highest AUC of 0.75 (0.73-0.77). Despite moderate model performances, specificity (5%) and precision (5%) were low. Most important features were: ‘absence of ADR in the Summary of product characteristics', ‘ADR reported as serious', ‘ADR labelled as an important medical event', ‘ADR reported by physician' and ‘positive rechallenge'. Model performances were similar when using only nine selected most important features.

Conclusions: We developed a prediction model to identify PSTRs and identified a set of most important features. Optimization of the model using more ICSR information (e.g. free text fields) to increase model precision is required before implementation.