(04) Limitations in evaluating effectiveness of Risk Minimization Measures: a qualitative analysis of industry sponsored post-authorization safety studies

Background: The European Medicines Agency (EMA) oversees safe and effective use of medicines throughout the product life cycle by, amongst others, implementation of risk minimization measures (RMMs). To assess effectiveness of these RMMs, post authorization safety studies (PASS) are performed by market authorization holders (MAHs). The Pharmacovigilance Risk Assessment Committee (PRAC) is mandated to evaluate outcomes of RMMs by assessing these PASS. Earlier studies show that concluding on RMM effectiveness is challenging, thus insights into aspects that limit these assessments are warranted.

Objectives: To describe limitations associated with RMM effectiveness assessments of industry sponsored PASS that did not render a conclusion by PRAC.

Methods: All PASS assessment reports finalized by PRAC between 1^st of January 2018 and 31^st of December 2021 were identified from PRAC minutes and compiled from non-public EMA databases (i.e., Documents Records Electronic Archive Management System and the European Review System for electronic Common Technical Documents). Text for analysis was extracted from PRAC’s final conclusion on the RMM effectiveness evaluation and sections in the assessment report dedicated to study limitations. A thematic analysis was conducted on extracted text using NVIVO software to identify (sub)themes presenting study limitations.

Results:

The study cohort consisted of 63 PASS, of which 61.9% were conclusive and 38.1% inconclusive  regarding evaluation of RMM effectiveness. For both conclusive and inconclusive PASS, the majority had a cross-sectional design with surveys as primary data source (74.4% and 70.8% respectively). Use of only secondary data was seen in 15.4% of conclusive and 37.5% of inconclusive PASS respectively.

               Four main themes emerged from the thematic analysis: (i) general limitations related to study design, (ii) survey design limitations, (iii) limitations related to use of secondary data, and (iv) limitations not related to study design. Limitations discussed in conclusive and inconclusive PASS were compared, and it was observed that some limitations were explicitly mentioned in inconclusive PASS, but not in conclusive PASS. Such limitations extracted from inconclusive PASS may also have been present in conclusive PASS, but were not explicitly discussed as limitation in the assessment reports of included conclusive PASS.

Limitations that were frequently explicitly mentioned in assessment reports of inconclusive PASS included, amongst others, limitations related to missing data (45.8%) and misclassification of data (16.7%). Additionally, participation bias (75.0% versus 59.0%), selection bias within the chosen database (41.7% versus 15.4%), and small sample in database (33.3% versus 15.4%) were more commonly explicitly mentioned in inconclusive compared to conclusive PASS.

Conclusions:

About 4 out of 10 PASS included in our study did not allow conclusion on RMM effectiveness. Well known common study limitations related to secondary use of data such as missing data were more often explicitly mentioned in inconclusive compared to conclusive PASS in our cohort. Results suggest need for strategies to mitigate limitations related to secondary use of data at protocol stage, e.g. through feasibility assessments. Although many databases may have incomplete registration of some variables, feasibility testing prior to conducting the PASS could contribute to meeting study objectives and allow conclusion on RMM effectiveness.