Systematic Reviews and Retractions: A Reflection on Quality

Objectives: Retracted publications continue to be cited in systematic reviews even after their retraction notices are issued. The presence of retracted publications in evidence synthesis prompts many questions about the impact of these retracted publications and the quality of the citing reviews. This study aims to assess the quality of systematic reviews and evidence synthesis citing retraction publications and the nature of those citations.

Methods: Building on previous research, we selected a set of 287 systematic reviews that cited retracted articles related to pharmacy, toxicology, and drug design after their retraction. Using a modified version of AMSTAR2, we assessed the quality of the citing systematic reviews and evidence syntheses. We also conducted a sentiment analysis to determine whether the retracted publication was cited positively, neutrally, or negatively in support of the findings of the systematic review. We will provide descriptive statistics outlining the quality of systematic reviews citing retracted research and investigate the association between the quality of a systematic review and the sentiment of the citation to a retracted publication.

Results: 70% (199/286) of the systematic reviews in our sample were found to be of critically low quality according to AMSTAR 2 criteria, while 17% (49) were low quality, 7% (21) were moderate quality, and 6% (17) were high quality. The 286 systematic reviews cited retracted research 324 times. Of these citations, 42.6% (138) were supportive, treating the retracted research as valid; 20.1% (65) refuted the findings or named the work as retracted (contrasting); and 37% (121) mentioned the research but neither supported nor contradicted it. We found statistically significant associations between quality and sentiment, with high quality studies being associated with contrasting citations and critically low quality studies being associated with supporting citations.

Conclusions: The prevalence of supportive citations to retracted research in systematic reviews is problematic. Associations between low methodological quality and uncritical use of retracted research may indicate that, although existing best practices may be effective in addressing retracted research in systematic reviews, the abundance of low quality systematic reviews that do not abide by these best practices may be problematic when we consider how the scientific literature corrects itself.