Emulation Differences versus Bias: Implications of Large-scale Emulation of Randomized Clinical Trials Using Clinical Practice Data (RCT-DUPLICATE)

Background: Regulators are interested in using real-world evidence (RWE) on the effectiveness of medical products to inform decision-making. While the utility for RWE to understand the safety of medications and inform health policy or practice has been recognized, the proliferation of studies with variable methodologic rigor makes it challenging for regulators to confidently make decisions on the effectiveness of drugs based on RWE studies.
RCT-DUPLICATE involved emulation of the design and analysis of 30 randomized clinical trials (RCT) using non-randomized data from clinical practice. The RWE study results were compared to the results of the RCTs in an effort to identify the circumstances under which causal conclusions can be reliably generated. Overall, the Pearson's (0.80) and intraclass correlation coefficients (ICC, 0.79 95% CI 0.59-0.89) were indicative of a strong correlation between RCT and RWE results. However, for a substantial minority of sampled trials, one or more pre-specified agreement metrics were not met. The narrow objective of RCT-DUPLICATE was to have a fair comparison between RCTs and RWE findings. This is best answered by focusing on the RCTs that could be emulated closely in terms of design and analytic strategy (14 RCTs, Pearson's 0.92; ICC 0.92 95% CI 0.66-0.96). Weak correlation between RCT and RWE results were observed for trials with substantive emulation challenges (16 RCTs, Pearson's 0.46; ICC 0.41 95% CI -0.03-0.73).
RWE studies come to the same conclusions as closely emulated RCTs when the data are fit-for-purpose. When divergence in findings is driven by emulation differences, the RWE study may be accurately targeting a different effect than the RCT.

Objectives: 1. Share the completed results from RCT-DUPLICATE's 30 trial emulations
2. Discuss emulation differences versus bias when comparing RCTs and RWE studies
3. Offer a platform for discussion between stakeholders representing different perspectives and the ISPE audience regarding the implications and lessons learned from trial emulations using clinical practice data.

Description: Dr. Shirley Wang will summarize emulation challenges and agreement metrics for RCT-DUPLICATE (30 min). Dr. Sebastian Schneeweiss will lead the discussion of results and will be joined by panelists with diverse stakeholder perspectives, including regulatory (Dr. Peter Arlett — EMA, 12 min), HTA/payer (Dr. Hans-Georg Eichler, 12 min) and industry (Dr. Melvin Olson, 12 min) The remaining time will be opened to the ISPE audience for Q&A (20 min).