(J) Machine Learning to Automate Data Extraction for Systematic Literature Reviews

Publication Number: 660

Background: Systematic literature reviews (SLRs) are routinely used to synthesise evidence from clinical trials and real-world studies/outcomes, but SLRs are time-intensive, with significant resources spent on data extraction. Automated data extraction using machine learning (ML) should enable timely completion of SLRs.

Objectives: To evaluate the use of ML to automate extraction of data from studies included in two SLRs, including randomized controlled trials in oncology (SLR 1) and observational studies in Fabry disease (SLR 2).

Methods: Using web-based tools, we annotated documents with terms of interest (entities) and relationships between them (relations). Articles identified from SLR 1 and SLR 2 were annotated alongside a larger set of similar articles. The process was two-stage: entity extraction – identification of entities; relation extraction – classification of entity relations.

Several models were trained and tested in their ability to identify entities and their relations vs human annotation, and the results compared. For entity extraction: BERT + CRF labels words in the context of neighbouring words. For relation extraction: BERT + Relation Classification, identifies relations between a pair of entities based on their location. BERT is pre-trained on general language text, so using biomedical literature, we also fine-tuned BERT to learn context-rich word representations. Performance was measured using relaxed F1 scores, a measure of specificity and sensitivity. We defined successful matches as partial overlap of entities of the same type.

Results: For entity extraction, biomedical fine-tuned BERT + CRF performed 4.1% better than BERT + CRF (F1 72.8% vs 68.7%) in SLR 1 and 8.8% better in SLR 2 (F1 69.5% vs 60.7%). Using fine-tuned BERT + CRF, in SLR 1, efficacy metrics were most robustly identified (F1 88.0%), while dosage was difficult to determine (F1 59.9%). In SLR 2, performance for patient age metric extraction was superior to the efficacy metric subgroup (F1 81.5% vs 44.1%, respectively). For relation extraction, the best performing model was fine-tuned BERT + Relation Classification with an F1 score of 94.6%. In cases with ≥ 80 label examples, F1 scores were at least 90% across all tested entity-type pairs.

Conclusions: BERT performance is enhanced when fine-tuned with biomedical literature and further improves when combined with a CRF model. With refinement, ML may be able to assist with human extraction of data for SLRs, substantially reducing the workload and decreasing the turnaround time for data synthesis.

Fully funded research: This research was supported by Millennium Pharmaceuticals, Inc. (a subsidiary of Takeda Pharmaceuticals).