(308) Evaluating The Efficacy of Clustering Methods Applied to Adverse Event Reports

Publication Number: 982

Background: VigiGroup is a cluster analysis algorithm developed at the Uppsala Monitoring Centre which aims to produce clinically coherent groups of adverse event (AE) reports to help overcome limitations with screening only single AE terms. However, a challenge with clustering methods is that they are difficult to evaluate. Therefore, a battery of clinically driven evaluation metrics has been developed, aimed at assessing the efficacy of clustering methods applied to AE reports.

Objectives: To design a set of clinically meaningful metrics aimed at assessing the performance of clustering methods applied to AE reports and to explore the value in the metrics.

Methods: A potential, improved clustering method was selected through a set of automated metrics. Firstly, it was computed how often the method was able to cluster together pairs of reports labelled as clinically related by a domain expert and separate reports labelled as unrelated. Secondly, it was measured whether known clusters could be recovered from artificially created datasets. Finally, the adjusted Rand index, a standard clustering metric, was calculated.

To further evaluate the proposed improved model against the original model, both were used to cluster an additional dataset. The resulting clusters were evaluated using an additional set of “Clinician-in-the-loop” metrics. These can be categorized into intruder detection metrics and in-depth metrics. Intruder detection quantifies the coherence of the clusters by measuring how well a domain expert can identify an intruding report or AE term added to a cluster. The in-depth assessment had domain experts attempt to label each cluster with a clinical theme. These were used to quantify the fraction of clusters with recognizable clinical themes, the extent to which clinical themes were repeated and how well the clusters covered the safety profile of the drug.

Results: The automated metrics showed that the modified version of vigiGroup was more effective in finding all the reports related to a clinical theme. However, the “Clinician-in-the-loop” metrics highlighted that the original method was better at creating coherent clusters and finding more distinct themes. The clinical impression from comparing the two methods aligned with the results of the in-depth assessment.

Conclusions: It was found that defining metrics driven by clinical expertise was invaluable in understanding the difference in the behavior of the methods and evaluating their practical applicability.