Highlights
- •Evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions.
- •Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time.
- •Lack of experience and domain knowledge, lack of collaborative and supportive software, as well as lack of good communication and management can increase resource use during the systematic review process.
Abstract
Objective
Study design and setting
Results
Conclusion
Keywords
Key findings: |
• Overall we identified 34 studies. Of these, 32 reported on resource use: three looked at resource use across the complete SR production process, while others only assessed the resource use of selected steps of the review process. Mostly, studies reported on the resource “time.” |
• Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time. |
• Four studies reported on reasons why steps are resource intensive: lack of experience and domain knowledge, lack of collaborative and SR-tailored software, doing steps manually instead of using supportive software, and lack of good communication and management of the SR process. |
What this adds to what is known: Our scoping review is the first to give an overview of the resource use during SR production. It shows that evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions. It also reveals that administration/project management is a time-consuming task that should be considered an important part of the SR process. |
What is the implication, what should change now: Based on our results, methods and tools to support project management and administration throughout a project, as well as methods and tools to speed up study selection, data extraction, and critical appraisal could help save resources. |
1. Introduction
Cochrane Community. Glossary [Available from: https://cdev.cochrane.org/glossary#letter-S.
- KQ 1. How many resources (e.g., time, costs) do different steps of the SR production consume?
- KQ 2. What are the reasons why some steps of the SR production are resource intensive?
2. Materials and methods
Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.
2.1 Study design
2.2 Information sources and search strategies
Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). in: aromataris e, munn z (editors). jbi manual for evidence synthesis, JBI, 2020 2020 [Available from: https://wiki.jbi.global/display/MANUAL/Chapter+11%3A+Scoping+reviews.
- 1)In a first step, an information specialist (IK) conducted a focused search of Ovid MEDLINE and Current Contents Connect (Web of Science) in November 2019. We screened these results to identify relevant studies for inclusion. Using PubReMiner and AntConc [[17],
Koster JA. PubReMiner 2014 [Available from: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi.
[18]], the included studies were analyzed to identify relevant text words contained in the title and abstract as well as Medical Subject Headings (MeSH) terms. - 2)Based on search terms derived from these included studies, IK developed a second, more comprehensive search strategy and searched the following databases in May 2020: Ovid MEDLINE, Scopus (Elsevier), Science Citation Index Expanded, Social Sciences Citation Index, and Current Contents Connect (all via Web of Science). The Ovid MEDLINE strategy was reviewed by a second information specialist (RS) in accordance with the Peer Review of Electronic Search Strategies (PRESS) guideline [[19]].
- 3)Using the studies identified by these searches, we conducted citation-based searches: manual screening of reference lists, forward citation tracking (via Scopus in May 2020), and a similar articles search (via PubMed, limited to the first 50 linked references for each seed article, in May 2020).
2.3 Eligibility criteria
Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].
Inclusion criteria | Exclusion criteria | |
---|---|---|
Topic | KQ1: Studies reporting the resource use of producing or updating SRs | Studies not reporting resource use |
KQ2: Studies reporting reasons why steps of the review production are resource intensive | ||
Concept | KQ1 and KQ2: Addressing the resource use of one or more steps of a SR (as depicted in Fig. 1) of health interventions or diagnostic or prognostic studies | Studies assessing other steps such as the dissemination of a SR's results |
Outcomes | KQ1: | Any other outcomes, such as time or money saved, without specification of the total resource amount spent on a task |
• Time spent on tasks (e.g., minutes per task) | ||
• Effort of personnel (e.g., full-time equivalent) | ||
• Costs (e.g., salary, license fees) | ||
• Material used | ||
KQ2: | ||
• Perceived reasons why some steps are/ are not resource intensive (barriers/ facilitators) | ||
• Perceived resource-intensive steps | ||
Study design/ publication type | KQ1: | Other publications (e.g., editorials, letters) |
• SRs | ||
• Empirical studies: measuring resource use by tracking | ||
• Surveys: asking about resource use | ||
• Simulation studies: modeling resource use | ||
KQ2: | ||
• Qualitative studies (e.g., interviews, open surveys) | ||
Full text availability | If full text was not retrievable via our libraries | |
Timing | 2009–2020 | |
Language | All languages |

2.4 Selection of sources of evidence
2.5 Data charting process and data items
2.6 Data synthesis
3. Results
3.1 Characteristics of the included studies
3.2 Comparative resource use across all steps of the systematic review process

3.3 Resource use during literature search
3.4 Resource use during study selection
3.5 Resource use during data extraction
3.6 Resource use during critical appraisal
3.7 Reasons for perceived resource intensity
4. Discussion
4.1 Limitations
5. Conclusion
Funding source
Author contribution
Conflict of interest
Acknowledgments
Appendix. Supplementary materials
References
- Expediting systematic reviews: methods and implications of rapid reviews.Implement Sci. 2010; 5: 56
Hartling L, Guise JM, Kato E, Anderson J, Aronson N, Belinson S, et al. Agency for healthcare research and quality (US). 2015:02.
- Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting.The Lancet. 2010; 376: 20-21
- Evidence-based research series-paper 1: what evidence-based research is and why is it important?.J Clin Epidemiol. 2021; 129: 151-157
- Evidence-based research series-paper 2: using an evidence-based research approach before a new study is conducted to ensure value.J Clin Epidemiol. 2021; 129: 158-166
- The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials.Trials. 2017; 18: 219
Cochrane Community. Glossary [Available from: https://cdev.cochrane.org/glossary#letter-S.
- Systematic review automation technologies.Syst. 2014; 3: 74
- It takes longer than you think: librarian time spent on systematic review tasks.J Med Libr Assoc. 2018; 106: 198-207
- Estimating time to conduct a meta-analysis from number of citations retrieved.Jama. 1999; 282: 634-635
Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.
- Scoping studies: towards a methodological framework.Int J Soc Res Methodol. 2005; 8: 19-32
- Scoping studies: advancing the methodology.Implement Sci. 2010; 5: 69
- Updated methodological guidance for the conduct of scoping reviews.JBI Evid Synth. 2020; 18: 2119-2126
- PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.Ann Intern Med. 2018; 169: 467-473
Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). in: aromataris e, munn z (editors). jbi manual for evidence synthesis, JBI, 2020 2020 [Available from: https://wiki.jbi.global/display/MANUAL/Chapter+11%3A+Scoping+reviews.
Koster JA. PubReMiner 2014 [Available from: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi.
- AntConc (Version 3.5.9) [Computer Software].Waseda University, Tokyo, Japan2020
- PRESS peer review of electronic search strategies: 2015 guideline statement.J Clin Epidemiol. 2016; 75: 40-46
The Cochrane Collaboration. Advances in evidence synthesis: special issue cochrane database of systematic reviews. 2020;(9 Suppl 1) 2020 [Available from: doi:10.1002/14651858.CD202001.
Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].
Clarivate. EndNote X8.
- Data extraction from machine-translated versus original language randomized trial reports: a comparative study.Syst. 2013; 2: 97
- Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.Agency for Healthcare Research and Quality (US), Rockville (MD)2013
- AHRQ Methods for Effective Health Care. Accuracy of Data Extraction of Non-English Language Trials with Google Translate.Agency for Healthcare Research and Quality (US), Rockville (MD)2012
- Evaluation of a new method for librarian-mediated literature searches for systematic reviews.Res. 2018; 9: 510-520
- Semi-automating the manual literature search for systematic reviews increases efficiency.Health Info Libr J. 2010; 27: 22-27
- A full systematic review was completed in 2 weeks using automation tools: a case study.J Clin Epidemiol. 2020; 121: 81-90
- Improving the translation of search strategies using the polyglot search translator: A randomized controlled trial.J Med Libr Assoc. 2020; 108: 195-207
- A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.Syst. 2017; 6: 234
- Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example.BMC Med Res Methodol. 2019; 19: 45
- A visual approach to validate the selection review of primary studies in systematic reviews: A replication study.in: Felizardo KR Barbosa EF Maldonado JC Proceedings of the International Conference on Software Engineering and Knowledge Engineering. SEKE, 2013
- Using visual text mining to support the study selection activity in systematic literature reviews.International Symposium on Empirical Software Engineering and Measurement;. 2011;
- Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.Inj Prev. 2019; 26: 26
- An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks.Methods Ecol Evol. 2019; (NIL_1-NIL_10)
Gresham G, Matsumura S, Li T. Faster may not be better: data abstraction for systematic reviews. Cochrane Colloquium; Hyderabad. AR I2014.
- Predicting the time needed for environmental systematic reviews and systematic maps.Conserv. 2019; 33: 434-443
- Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.PLoS ONE. 2011; 6: 6
- Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.J Clin Epidemiol. 2015; 68: 191-199
- Opportunities for computer support for systematic reviewing - a gap analysis.Transform Digit Worlds. 2018; 10766 (2018): 367-377
- Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.J Clin Epidemiol. 2010; 63: 289-298
- Extracting data from figures with software was faster, with higher interrater reliability than manual extraction.J Clin Epidemiol. 2016; 74: 119-123
- Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.J Clin Epidemiol. 2020; 128: 140-147
- Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity.J Clin Epidemiol. 2013; 66: 408-414
- Identifying and removing duplicate records from systematic review searches.J Med Libr Assoc. 2015; 103: 184-188
- A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.J Clin Epidemiol. 2019; 115: 77-89
- Survey of systematic review authors in dentistry: challenges in methodology and reporting.J Dent Educ. 2009; 73: 471-482
- Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.BMC Med Res Methodol. 2017; 17: 152
- An exploration of crowdsourcing citation screening for systematic reviews.Res. 2017; 8: 366-386
- Crowdsourcing the citation screening process for systematic reviews: validation study.J Med Internet Res. 2019; 21: e12953
- Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities.JMIR Med Inform. 2014; 2: e11
- Improving the conduct of systematic reviews: a process mining perspective.J Clin Epidemiol. 2018; 103: 101-111
- Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses.J Clin Epidemiol. 2019; 105: 92-100
- Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.Evid Based Libr Inf Pract. 2014; 9: 28-50
- AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.J Clin Epidemiol. 2009; 62: 1013-1020
- Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.Syst. 2016; 5: 140
- Dual computer monitors to increase efficiency of conducting systematic reviews.J Clin Epidemiol. 2014; 67: 1353-1357
- Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.Evidence Based Library and Information Practice. 2019; 14: 80-83
- Citation searching: a systematic review case study of multiple risk behaviour interventions.BMC Med Res Methodol. 2014; 14: 73
- Usage of automation tools in systematic reviews.Res. 2019; 10: 72-82
Article info
Publication history
Identification
Copyright
User license
Creative Commons Attribution (CC BY 4.0) |
Permitted
- Read, print & download
- Redistribute or republish the final article
- Text & data mine
- Translate the article
- Reuse portions or extracts from the article in other works
- Sell or re-use for commercial purposes
Elsevier's open access license policy