(7) Research Impact Assessment: An Example from The Canadian Network for Observational Drug Effect Studies

Publication Number: 338

Background: The Canadian Network for Observational Drug Effect Studies (CNODES) evaluates the type, magnitude, and frequency of risks and benefits that are associated with drug use after marketing. CNODES also builds data, methods, and human resource capacity to conduct drug safety and effectiveness research and evaluate its impact.

Objectives: The goal was to have a framework to document and assess the research impact of CNODES.

Methods: We used a theory-driven developmental evaluation approach as CNODES is a large, geographically dispersed research network with many stakeholders, and it exists in a changing drug regulatory and policy environment. Our utilization-focused approach takes into consideration diverse perspectives and intended uses of the assessment. We used the Theoretical Domains Framework version 2, with 14 domains. We analyzed the bibliometric and altmetric data for 95 CNODES publications.

Results: Our outputs included a Logic Model and Theory of Change to guide CNODES research impact assessment. These outputs 1) identified research impact activities and outputs; 2) identified reach, including key academic and government stakeholders; 3) used the Theoretical Domains Framework to identify assumptions related to stakeholders’ use of CNODES research; and 4) identified examples of desired behavior changes/changes in practice; for example, decision-makers rely on and use CNODES evidence to help inform drug safety and effectiveness policies and practices. Using this framework, we adapted bibliometric and altimetric tools to document research impact. One citation indicator showed that 24% of CNODES publications are among the top 5% most cited in their field, with the average download per publication being 35,003 (IQR 8,043-45,047).

Conclusions: CNODES developed a logic model, theory of change, and evaluation matrix to guide integration of research impact assessment into the CNODES research process. CNODES identified stakeholders and considered their capacity, opportunity, and motivation for using CNODES research to inform decisions. CNODES used bibliometrics, altmetrics, media analysis, and authorship gender analysis to assist with impact documentation and is now automating these processes. These approaches provide for the ongoing assessment of ways in which CNODES research is being used by diverse stakeholder groups.
Acknowledgment
Canadian Network for Observational Drug Effect Studies (CNODES), a collaborating centre of the Drug Safety and Effectiveness Network (DSEN), is funded by the Canadian Institutes of Health Research (CIHR, Grant #DSE – 146021).