Find the Studies Without the Noise: Validation of the Performance of a Pragmatically-Designed PubMed Methodological Filter

Objectives: Comprehensive review searches can generate a high volume of results, including many non-empirical publications that present no original patient/participant research data and will be screened out during the title/abstract review phase. Using a set of included studies from previously published systematic reviews (SRs), we demonstrate the relative recall performance of a pragmatically-designed PubMed search filter, the Non-Empirical Publication (NEP) filter, intended to distinguish and exclude non-research publications from comprehensive searches and still retrieve publications of interest.

Methods: We identified 32 SRs published in 2017 and indexed in PubMed which reported their included studies. The focus of the SRs included treatment, epidemiology, diagnosis, and other. The SRs included studies of a broad range of methodologies, including RCTs, non-randomized clinical trials, observational studies, and case reports. We measured the ability of the NEP filter to retrieve the Pubmed-included studies included in these reviews.

Results: The 32 SRs included a total of 1172 studies, of which 1062 were found in PubMed by known-item searching. The NEP filter demonstrated a relative recall of 0.98 across all 32 reviews. Relative recall for SR subsets was as follows: Treatment SRs - 0.97; Epidemiology SRs - 0.98; Diagnosis SRs - 0.98; Other SRs - 0.99

Conclusions: The NEP filter performs very well in retaining studies of interest using a relative recall method. It holds its high recall rate across SRs of different types and across included studies of a broad range of methodologies. Work is continuing to establish sensitivity, precision and reduction in number needed to read.