Multiple Probabilistic Automated Publication Type Taggers to Index Biomedical Articles and Support Evidence Synthesis

Objectives: Information specialists have developed various search hedges and filters in order to retrieve studies with different publication types and study designs (e.g., randomized controlled trials, cohort studies, case control studies, etc.) for information retrieval. This is particularly important for evidence synthesis which requires comprehensive retrieval. Manual indexing is neither rapid enough, accurate enough, nor consistent enough across bibliographical databases. Therefore, we trained a set of automated machine learning based probabilistic taggers for 50 publication types and study designs. Probabilistic scores are more flexible than binary yes/no tags, and have thresholds that can be tailored for specific use cases.

Methods: Training data was based on human-related PubMed article records from 1987-2014. Title and MEDLINE/PubMed information such as MeSH terms and MeSH Publication Types were used to create labeled training sets. Predictive features included title and abstract bigrams, journal similarity, and various measures of pairwise similarity across two articles (e.g., important word scores, implicit term similarity scores, and Paragraph2Topic vectors). Taggers were trained using liblinear SVM, recalibrated on 2015 PubMed data, and then tested on manually assigned PubMed publication type data from the years 2016 and 2019. We measured performance with the area under the curve of the Receiver Operating Characteristic (AUC-ROC), which measures the tradeoff between maximizing true positives and minimizing false positives. We also carefully reviewed manually several of the most important publication types for systematic review, including Cohort Studies, Cross-Over Studies, Cross-Sectional Studies, and Randomized Controlled Trials.

Results: Overall the AUC-ROC performance is very high: The best performing tagger was TWIN_STUDY with an AUC of 0.996, and most of the taggers have AUC values above 0.9. For Cohort Studies, Cross-Over Studies, Cross-Sectional Studies, and Randomized Controlled Trials, when articles lacked NLM indexing but were strongly predicted as such by our model, manual review verified that the model was correct in the majority of cases.

Conclusions: We have created a dataset for all PubMed articles having abstracts of their probabilistic prediction scores for 50 Publication Types and study designs. These will be made publicly available for download at https://arrowsmith.psych.uic.edu/evidence_based_medicine/index.html. These have also been incorporated into the Anne O’Tate tool https://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/AnneOTate.cgi: When a user makes a PubMed query, the list of retrieved articles are displayed according to both NLM-indexed Publication Types as well as our predicted publication types and study designs. Publication type taggers may be of particular interest for systematic review teams and in resource-constrained settings such as rapid reviews.