Analysis of Public Preprint Server Comments on NIH Preprint Pilot Articles

Objectives: Given the increased prevalence of preprints during the COVID-19 pandemic, this project sought to analyze public comments left on a sample of preprint articles from the NIH Preprint Pilot to determine if they were substantive in nature. Analysis of article titles and qualitative coding of the comments was conducted. This analysis was designed to obtain both quantitative and qualitative measures of comments on a selected group of articles so that the relationship between public commenting and scientific rigor could be explored.

Methods: The first 1,000 preprint articles to be indexed in PubMed Central and hosted on two preprint platforms (bioRxiv and medRxiv) were selected. Using the preprint servers’ associated commenting platforms, full text comment threads and Twitter information was obtained, and summary statistics of commenting platforms were produced. From the article sample a total of 494 comments were collected from public commenters using the Disqus platform to provide feedback on the articles. Using the article titles, the authors explored indications of the relationship between article topic and frequency of commenter engagement. Preliminary coding was conducted using a ‘thumbs up/thumbs down’ method and potential categorizations were suggested. Utilizing these suggestions, the authors created and refined a draft codebook. Finally, thirteen categorizations, ten for substantive comments and three for not substantive comments, were created and used to qualitatively code the comment sample.

Results: Two rounds of coding were completed to reach sufficient interrater reliability. The authors found that most of the public comments were substantive, with over 28% meeting the criteria for critique, 21.5% as questions for authors, and over 11% having aspects of a formal peer review process. The analysis revealed engagement between commenters and preprint authors, demonstrated through author responses to questions, updates, and feedback. Commenters also provided suggestions for future research (3.6%) and indicated their intent to utilize the preprint findings in future research projects of their own (2.6%).

Conclusions: This project provides evidence of the impact of public commenting on scientific rigor. Public commenting was frequently substantive, and provided critique which sometimes led to direct revisions of the preprint article. Commenters also provided responses similar in nature to the formal peer review process, providing authors with feedback faster than the traditional process. Through preprints authors can disseminate their research to a wide audience earlier, and comments indicated that some readers intended to use the preprint findings in their own research, accelerating the potential for scientific discovery. As the prevalence of preprints continues to grow and public engagement with preprints increases, this paper’s methodology can be replicated and refined to further analyze the value of public commenting on preprints.