Transforming Evidence: Measuring the Prevalence of Preprint Inclusion in Systematic Reviews

Objectives: In the wake of COVID-19, submission of health sciences preprints has grown exponentially. While grey literature should be included in any comprehensive systematic review, some preprint servers emphasize that preprints should not be relied on to guide clinical practice. This could be considered in direct opposition to the goal of evidence synthesis. Considering the rigor of a systematic review and meta-analysis, this exploratory study aimed to discover how often preprints are included in systematic reviews.

Methods: PubMed, Web of Science, the JBI Library, and the Cochrane Library were searched from inception to April 2021 for systematic reviews that included bioRxiv, medRxiv, or ChemRxiv in their information sources. Systematic review protocols were not included in the search. Results were imported into EndNote to remove duplicates and errata. The bibliographies of the remaining citations were searched for inclusion of preprint citations in the final review. Results were then analyzed by discipline, journal, country, and type of review to determine any publication trends.

Results: A total of 212 systematic reviews were included in the analysis. The reviews were published from 2019-2021 and 98% were related to COVID-19. A sample of the results indicates that general internal medicine was the most common discipline with 39% of the citations. Out of 154 publications, PLOS One was represented the most frequently (seven reviews), and six of the reviews were published in medRxiv. Reviews with at least one author from the United States were the most common (32%), and 85% of the citations followed the methodology of a quantitative systematic review. Other systematic review methodologies included living, rapid, scoping, and, remarkably, one umbrella review.

Conclusions: Owing to medRxiv’s relatively recent inception and its role in pushing COVID-19 research forward, the number of reviews concerning COVID-19 is to be expected. While most of the analyzed reviews included only one or two preprint citations, several included a significant number of preprints. When accounting for the lack of evidence published through traditional channels during the COVID-19 pandemic, the synthesis of preprints is understandable; however, basing clinical practice off of non-refereed publications should be done with an abundance of caution. As this was an exploratory analysis, future data collection would certainly capture reviews beyond COVID-19 and offer further insight into the impact of preprint inclusion in evidence synthesis.