Background and Objectives
- •Abstrackr-assisted screening resulted in title and abstract screening workload savings when used to reduce the number of citations to be screened in a systematic literature review of treatments for relapsed/refractory diffuse large B cell lymphoma.
- •These workload savings did not come at the expense of omitting relevant citations.
- •The performance of Abstrackr's text-mining functions has varied in the literature. This study adds to the evidence base by using a complex systematic literature review, with a diverse range of interventions and study types. Inclusion and exclusion criteria had lexical similarity, adding a further dimension of complexity.
- •Workload savings from the perspective of a national decision-making agency have been estimated.
What this adds to what is known?
- •The evidence base of text-mining tools is limited. Further research is required to assess the performance of Abstrackr's text-mining functions at higher maximum prediction scores (i.e., earlier stopping points). Generalizability of the results presented here may be limited. Thus, further research is also required to determine if Abstrackr-assisted screening can be considered in other research questions and disease areas.
What is the implication, what can we change now?
1.1 Challenges in title and abstract screening
- Hearst M.
1.3 Text-mining tool: Abstrackr
Machine leaning functionality in EPPI-Reviewer 2021.
|Sensitivity (true positive rate)||The proportion of citations correctly predicted as relevant by Abstrackr out of the total deemed relevant by the human screener [|
|Specificity (true negative rate)||The proportion of citations correctly predicted as irrelevant by Abstrackr out of the total deemed irrelevant by the human screener [|
|Precision||The proportion of citations correctly predicted as relevant by Abstrackr amongst all citations predicted as relevant by Abstrackr (both correct and incorrect)|
|False negative rate||The proportion of citations that were incorrectly predicted as irrelevant by Abstrackr, out of the total number of citations deemed relevant by the human screener|
|Proportion missed||The proportion of citations incorrectly predicted as irrelevant by Abstrackr that were included in the final evidence base, out of the total number of citations included in the final evidence base|
|Workload savings||The proportion of citations predicted as irrelevant by Abstrackr out of the total number of citations to be screened (i.e., the proportion of citations that would not need to be screened manually)|
|Time savings||Time saved based on the citations that would not need to be screened (i.e., those predicted as irrelevant by Abstrackr); estimated based on a screening rate of 0.5 min per citation and an 8-hour work day|
2. Aim of study
3.1 Choice of data set
3.2 Search methods
3.3 Citation management
3.4 Abstrackr-assisted screening
3.4.1 Sensitivity analysis
3.5 Single-human screening: Covidence
3.6 Data analysis to assess the performance of Abstrackr's text-mining functions
4.1 Performance of Abstrackr's text-mining functions
|Human screener (Screener 2) judgments|
|False negative rate|
|Result (prediction score 0.39540)||91||72||15.5||9||0||67||5.4|
|Result (prediction score 0.34458)||97||64||13||3||0||59||4.8|
|Result (prediction score 0.29021)||100||54||11||0||0||50||4.0|
4.1.1 Sensitivity analysis
5.1 Main findings
CRediT authorship contribution statement
Appendix ASupplementary Data
- Appendix A and B
- Cochrane Handbook for Systematic Review of Interventions version 6.2. John Wiley & Sons, Chichester (UK)2021
- Following 411 Cochrane protocols to completion: a retrospective cohort study.PLoS One. 2008; 3: e3684
- Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool.Syst Rev. 2018; 7: 45
- Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry.BMJ Open. 2017; 7: e012545
- Evidence summaries: the evolution of a rapid review approach.Syst Rev. 2012; 1: 10
- Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers.Syst Rev. 2015; 4: 80
- Using text mining for study identification in systematic reviews: a systematic review of current approaches.Syst Rev. 2015; 4: 5
- Text mining for biology and biomedicine.Artech House, Boston/London2006
- Untangling text data mining.37th Annual Meeting of the Association for Computational Linguistics, 1999https://dl.acm.org/doi/10.3115/1034678.1034679Date accessed: June 10, 2022
- Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer.Syst Rev. 2020; 9: 73
- The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews.BMC Med Res Methodol. 2020; 20: 139
- Rayyan—a web and mobile app for systematic reviews.Syst Rev. 2016; 5: 210
- Machine leaning functionality in EPPI-Reviewer 2021.(Available at)https://eppi.ioe.ac.uk/CMS/Portals/35/machine_learning_in_eppi-reviewer_v_7_web_version.pdfDate accessed: July 12, 2021
- Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools.Syst Rev. 2019; 8: 278
- Deploying an interactive machine learning system in an evience-based practice center: abstrackr. 2nd ACM SIGHIT International Health Informatics Symposium.Association for Computing Machinery, New York2012
- Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews.Syst Rev. 2020; 9: 272
- Abstrackr: online, open-access, free software for citation screening. KTDRR and Campbell collaboration research evidence training session.Centre on Knowledge Translation for Disability & Rehabilitation Research, Austin, TX2019
- Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence.Syst Rev. 2020; 9: 243
- Diagnostic tests. 1: Sensitivity and specificity.British Medical Journal. 1994; 308
- Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.Syst Rev. 2016; 5: 140
- KBA: Kernel boundary alignment considering imbalanced data distribution.IEEE Trans Knowl Data Eng. 2005; 17: 786-795
- Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial.J Clin Epidemiol. 2020; 121: 20-28
Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Data availability statement: The data that support the findings of this study are available from the corresponding author upon reasonable request.
Declarations of interest: None declared by any of the authors.
User LicenseCreative Commons Attribution (CC BY 4.0) |
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy