If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Department of Health Systems Management, Guilford Glazer Faculty of Business and Management and Faculty of Health Sciences, Ben-Gurion University of the Negev, IsraelInstitute of Health Policy Management and Evaluation, Dalla Lana School Of Public Health, University of Toronto, Canada
National School of Public Health, Management and Professional Development Bucharest, RomaniaSpiru Haret University, Faculty of Psychology and Educational Sciences
Department of Clinical Pharmacology, Cantonal Hospital Zenica, Zenica, Bosnia and HerzegovinaFaculty of Medicine, University of Zenica, Zenica, Bosnia and Herzegovina
Centro de Análisis de la Evidencia Científica, Academia Española de Nutrición y Dietética, EspañaTechné research group. Department of knowledge engineering of the Faculty of Science. University of Granada. Spain
Cochrane Russia at the Russian Medical Academy for Continuing Professional Education (RMANPO) of the Ministry of Health of Russian Federation and the Kazan State Medical University of the Ministry of Health of Russian Federation
Cochrane Austria, Danube University Krems, Krems a.d. Donau, AustriaDepartment of Family Medicine, Care and Public Health Research Institute (CAPHRI), Maastricht University, The Netherlands
Cochrane Netherlands, Julius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the NetherlandsAmsterdam UMC, Univ of Amsterdam, Amsterdam Public Health, Medical Library, Meibergdreef 9, Amsterdam, Netherlands
Nutrition & Metabolism, NOVA Medical School, Faculdade de Ciências Médicas, Universidade NOVA de Lisboa, Campo dos Mártires da Pátria, 1169-056 Lisboa, PortugalCINTESIS, NOVA Medical School, NMS, Universidade Nova de Lisboa, Campo dos Mártires da Pátria, 1169-056 Lisboa, Portugal
Evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions.
•
Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time.
•
Lack of experience and domain knowledge, lack of collaborative and supportive software, as well as lack of good communication and management can increase resource use during the systematic review process.
Abstract
Objective
We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.
Study design and setting
We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.
Results
We included 34 studies. Thirty-two reported on the resource use—mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.
Conclusion
Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.
• Overall we identified 34 studies. Of these, 32 reported on resource use: three looked at resource use across the complete SR production process, while others only assessed the resource use of selected steps of the review process. Mostly, studies reported on the resource “time.”
• Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time.
• Four studies reported on reasons why steps are resource intensive: lack of experience and domain knowledge, lack of collaborative and SR-tailored software, doing steps manually instead of using supportive software, and lack of good communication and management of the SR process.
What this adds to what is known: Our scoping review is the first to give an overview of the resource use during SR production. It shows that evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions. It also reveals that administration/project management is a time-consuming task that should be considered an important part of the SR process.
What is the implication, what should change now: Based on our results, methods and tools to support project management and administration throughout a project, as well as methods and tools to speed up study selection, data extraction, and critical appraisal could help save resources.
Well-conducted systematic reviews (SRs) are considered the most reliable form of evidence syntheses because they employ high methodological standards in summarizing primary research. SRs play an important role as support for evidence-based clinical and health policy decision-making. However, conducting a SR is very resource intensive and can take up to 2 years for completion [
]. This often does not meet the needs of decision-makers, especially in times where evidence syntheses must answer urgent questions such as during the ongoing coronavirus pandemic.
SRs are also essential in primary research. Systematically reviewing the existing evidence before starting a new study is important to ensure high quality and relevant primary research [
]. Knowing all the studies in a field helps focus on topics and research questions that require new studies. In addition, learning from earlier studies helps optimally design new studies [
]. Ideally, a comprehensive up-to-date SR informs every new study. However, as the methodological standards in SR production are very high, the steps to develop or update a SR are complex and resource intensive. This can keep primary researchers from conducting or updating a SR [
According to Cochrane, a SR is “a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyses data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyses and summaries the results of the included studies” [
]. To conduct a SR, certain steps must be completed, from formulating a focused research question to conducting a comprehensive search strategy, screening the identified literature, and critically appraising and synthesizing the primary studies [
]. All these steps require resources such as time or money to complete. Performing a high-quality systematic search, for example, requires an information specialist's expertise and time. A recent study showed that information specialists needed an average aggregated time of 26.9 hours when developing a search strategy [
]. Another study assessed the resource need for completing a SR with meta-analyses as ranging from 216 to 2518 hours, depending on the number of studies included and the comparisons and outcomes assessed [
The production and update of SRs must become more resource efficient to meet the time-sensitive needs of clinical and health policy decision-makers and to facilitate the uptake of an evidence-based research approach by primary researchers. To discover where the largest gain in improving efficiency might be possible, we wanted to map the resource use of different SR steps. This could help identify the most resource-intensive areas in the review process. In addition, we wanted to know the reasons why certain steps of the review process are resource intensive, in order to identify suitable methods to address them. Therefore, we conducted a scoping review of the published literature mapping the resource use of each step of the SR process and the reasons why steps of the SR process are resource intensive. Specifically, we strove to answer the following two key questions (KQs):
KQ 1. How many resources (e.g., time, costs) do different steps of the SR production consume?
KQ 2. What are the reasons why some steps of the SR production are resource intensive?
2. Materials and methods
This scoping review is part of working group 3’s work within the EVBRES (EVidence-Based RESearch) – COST Action CA17117, funded by the European Union (www.evbres.eu). The protocol for this scoping review was published a priori via the Open Science Framework (https://osf.io/fby54/) [
Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.
We conducted a scoping review. Within EVBRES, we define a scoping review as a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, theories, types and sources of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge. We followed the guidance given by Arksey and O'Malley [
]. We reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews (PRISMA-ScR) [
In a first step, an information specialist (IK) conducted a focused search of Ovid MEDLINE and Current Contents Connect (Web of Science) in November 2019. We screened these results to identify relevant studies for inclusion. Using PubReMiner and AntConc [
], the included studies were analyzed to identify relevant text words contained in the title and abstract as well as Medical Subject Headings (MeSH) terms.
2)
Based on search terms derived from these included studies, IK developed a second, more comprehensive search strategy and searched the following databases in May 2020: Ovid MEDLINE, Scopus (Elsevier), Science Citation Index Expanded, Social Sciences Citation Index, and Current Contents Connect (all via Web of Science). The Ovid MEDLINE strategy was reviewed by a second information specialist (RS) in accordance with the Peer Review of Electronic Search Strategies (PRESS) guideline [
Using the studies identified by these searches, we conducted citation-based searches: manual screening of reference lists, forward citation tracking (via Scopus in May 2020), and a similar articles search (via PubMed, limited to the first 50 linked references for each seed article, in May 2020).
The database searches were limited to studies published since 2009 (the year when PRISMA was published). Last, to identify grey literature, we contacted experts in the field and screened the latest proceedings of the Cochrane Colloquium [
] to also include new studies that might not have been published yet. Details on the search strategies are available in web appendix 1.
2.3 Eligibility criteria
The eligibility criteria are specified in Table 1. For KQ 1, we included studies that assessed the resource needs of one or more steps of the review process. Studies that tracked resource use when conducting SRs and those that asked for resource use via a survey as well as studies that modeled resource use were eligible. If a study only reported relative measures of resource use, such as time or workload saved, without specifying the absolute resource use of the task of interest, the study was excluded. To define the SR production steps, we used the list of steps provided by Tsafnat et al. 2014 [
Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].
]. When mapping the results, we identified an additional SR production step and added “Administration/project management” a posteriori (Fig. 1). For KQ 2, we were interested in qualitative studies that asked SR authors via interviews or surveys why some steps of the SR production are resource intensive. We limited the publication date to 2009 because the current reporting standards, the PRISMA guidelines, were published at that time.
Table 1Eligibility criteria.
Inclusion criteria
Exclusion criteria
Topic
KQ1: Studies reporting the resource use of producing or updating SRs
resource use must already be mentioned in the abstract
Studies not reporting resource use
KQ2: Studies reporting reasons why steps of the review production are resource intensive
Concept
KQ1 and KQ2: Addressing the resource use of one or more steps of a SR (as depicted in Fig. 1) of health interventions or diagnostic or prognostic studies
Studies assessing other steps such as the dissemination of a SR's results
Outcomes
KQ1:
Any other outcomes, such as time or money saved, without specification of the total resource amount spent on a task
• Time spent on tasks (e.g., minutes per task)
• Effort of personnel (e.g., full-time equivalent)
• Costs (e.g., salary, license fees)
• Material used
KQ2:
• Perceived reasons why some steps are/ are not resource intensive (barriers/ facilitators)
• Perceived resource-intensive steps
Study design/ publication type
KQ1:
Other publications (e.g., editorials, letters)
• SRs
• Empirical studies: measuring resource use by tracking
• Surveys: asking about resource use
• Simulation studies: modeling resource use
KQ2:
• Qualitative studies (e.g., interviews, open surveys)
Full text availability
If full text was not retrievable via our libraries
After piloting the abstract screening with 50 records, the author team used Covidence (www.covidence.org) to dually and independently screen records based on titles and abstracts and full texts. We resolved any screening discrepancies at the abstract or full-text level by discussion or by consultation with a third author. We stored records in the reference management software EndNote [
We developed and piloted a data extraction form using Google Forms. Two team members independently extracted data; a third author made final decisions in cases of discrepancies.
2.6 Data synthesis
We mapped the results of the scoping review in a summary table, grouping them along the steps of the review process. Because the goal of this scoping review was to descriptively map the resource use of SR production steps rather than to derive cause–effect relationships, we did not apply a formal certainty of evidence or risk of bias (RoB) assessment.
3. Results
We included 34 studies (32 quantitative primary studies, 1 qualitative primary study, 1 SR) that were published in 38 publications [
Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
Evidence Based Library and Information Practice.2019; 14: 80-83
] (Fig. 2: PRISMA study flow). In the following sections, we first summarize the characteristics of the included studies. We then present the resource use for single steps of the review process as well as across all the steps combined. Finally, we summarize the reasons why the review authors perceive some steps of the SR process as resource intensive.
3.1 Characteristics of the included studies
Thirty-two studies contributed to KQ1 and reported on resource use, mostly on time spent [
Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
Evidence Based Library and Information Practice.2019; 14: 80-83
], while the other 29 focused on one or more single steps: study selection (title/abstract screening n = 7, obtaining full-text articles n = 1, full-text screening n = 4), literature search (n = 11), data extraction (n = 7), and critical appraisal (n = 4). Four studies helped answer KQ2 and described reasons why certain steps of the review process are resource intensive [
Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
]. In web appendix 2, we summarize the characteristics of all the included studies. In web appendix 3, we map the resource use to the steps of the SR process.
3.2 Comparative resource use across all steps of the systematic review process
Three studies assessed the resource use required to conduct complete SRs [
] simulated SR production and estimated the resource needs for each step based on survey responses from SR authors or on the literature. According to Pham et al., the mean time needed to conduct all SR tasks was 881 person-hours, ranging from 243 to 1752 person-hours, depending on the scope of the SR. When comparing different steps of the review process, protocol development and literature search were less resource intensive than selecting studies and extracting data, which together took roughly 50 % of all the time resources needed [
]. Haddaway et al. simulated that the average time to conduct a SR is 1312 person-hours. They also concluded that study selection as well as data extraction and critical appraisal of studies are very time consuming. Searching, retrieving full texts, and assembling a library of relevant studies required less time than most stages. The largest proportion of time, however, was needed for administration and planning [
] published the only empirical study tracking the time needed to conduct one SR with the help of automation tools. Their SR identified 1694 abstracts and finally included 8 publications. They completed it in 61 person-hours. The least time-consuming tasks were searches, deduplication, and obtaining full texts. The most time-consuming steps were administration and project management, abstract and full-text screening, data extraction, critical appraisal, and data synthesis [
]. Studies also reported the resource use for different literature search components. Developing a search strategy for a SR took 1 to 13.5 hours, depending on the software support (text mining) used [
]. Translating a search strategy for another database took 11 to 79 minutes manually and 6 to 57 minute per database using Polyglot Search Translator [
]. Deduplication of records was estimated to take 3 to 10 minute per SR, depending on the software used; however, the authors also report deduplicates missed by the software that required manual reduplicating later in the process [
Studies reported the resource use of different additional search approaches: manual reference list checking of included studies can take 8 hours compared to 3 hours when using Scopus [
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
]. Conflict resolution–often necessary in dual screening processes–took on average 5 minute per conflict, and retrieving full-text articles took 4 minutes per full text [
Different approaches to title/abstract screening exist, and although they can reduce the time needed for this step, they may also increase the full-text screening burden. Shemilt et al. [
] analyzed the following approaches: (1) single screening; (2) single screening with semiautomated software (one person screens abstracts and automatically excludes those deemed irrelevant by the algorithm); (3) dual screening (two people independently screen all abstracts and discuss conflicting decisions); and (4) safety first (two people independently screen all abstracts, all potentially relevant abstracts are included for full-text screening). These approaches were all followed by dual independent full-text screening. Based on a project with 12477 abstracts, the semiautomated approach took the least time (572 hours) and cost the least (£ 37,860), while dual screening took twice as long (1089 hours) and cost nearly twice as much (£ 75139) [
Two studies assessed a type of crowdsourcing for title/abstract screening. While experts cost between $ 3034 and $ 8777 to complete such a task, the crowd costs ranged between $ 458 and $ 2223. However, it required the crowd 4 to 17 days to complete the screening [
One study showed that diagnostic test accuracy (DTA) reviews have on average 185 % more workload during abstract screening and 167 % more during full-text screening, because searches for DTA reviews identify many more records [
A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
Sometimes the SR team must translate a study into English before being able to extract data. Using Google Translate took on average 15 to 60 minutes per study, depending on the publication's original language (Spanish: 15 minutes to Chinese: 60 minute) [
Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
]. Applying ROBINS-I plus finding consensus took the longest, at 48.45 minutes, while applying ROB-NRSE and reaching consensus took on average 36.98 minutes [
Facilitators: Methodological experience and content knowledge of team members, existing data extraction sheets from former projects, blocking time for the SR, daily project meetings to discuss upcoming questions, and writing the protocol in past tense all contributed to speeding up the process. In addition, team members’ physical proximity allowed for ongoing communication and short time lapses between tasks, and familiarity with the used software tools helped [
Barriers: Lack of domain expertise, juggling other projects with competing deadlines, noisy surroundings, resource unavailability, poor internet, and software incompatibilities and limitations (e.g., automation of only one task) increased the resource use [
]. Hoang et al. also identified a gap between the available SR support tools and the tools used. The majority of review authors still rely only on Excel and EndNote, not using other SR-tailored tools. Potential explanations were that the reviewers are not aware of the other tools, might have limited access due to costs, or feel more comfortable using well-known tools [
]. Balk et al. highlighted that automated translation needs more resources for non-European languages and tables than for European languages, and if manuscripts are not available in Word or PDF formats [
To the best of our knowledge, this is the first scoping review mapping the resource use required to conduct a SR. Across all SR production steps, study selection, data extraction, and critical appraisal seem to be very resource intensive while protocol development, literature search, and study retrieval take less time [
]. Project management and coordination needed the largest proportion of SR production time. This is relevant for future initiatives that aim to make the SR production process more efficient, since this task is usually not in focus when thinking about SR production steps.
We did not identify any study reporting on how much resources the certainty of evidence assessments requires, and only 6 studies reported resources other than time. We also did not identify a study focusing specifically on updating an existing SR; however, the resource use for individual steps of the SR process is probably similar when updating a review. The only study reporting resource use for the specific steps of formulating the review question, searching for existing SRs, writing a protocol, synthesis/ meta-analysis, and writing up the report was that of Clark et al. [
]. This study was a case study of a single SR, so the generalizability is very limited.
Our scoping review showed that literature search requires only a small proportion of the overall time required to complete a SR. This contradicts survey results among scientists in dentistry who perceived literature search as a particularly time-consuming challenge [
]. In the included studies of our scoping review, information specialists performed the search steps, which might explain why literature search did not take so much time though it is a very complex step in the SR process.
The time needed for study selection, data extraction, and critical appraisal varied largely depending on the number of identified records and full texts. Other factors such as lack of experience or domain expertise might also increase resource use. In addition, the lack of using collaborative and supportive software increased the resource need [
]. Clark 2020a highlighted that the least time-consuming SR production tasks were generally those where the most automation tools were available, and vice versa [
Our scoping review has several limitations. First, we focused only on the resource use of different steps and approaches, not on the validity and accuracy of the methods or tools. We plan to answer this in a follow-up scoping review (https://osf.io/9423z). Second, we did not assess the RoB of the included studies. Instead, we described the methodological approach of the included studies for transparency. Third, we limited the inclusion criteria to studies published from 2009 onward—the year when PRISMA was published. We might have missed older but informative and relevant studies. However, we think that the included studies provide a good overview of current resource use in SR production. Fourth, we included only studies that mention any type of resource use in the abstract. We considered this necessary to achieve a balance of sensitivity and specificity during the screening process. Fifth, the included studies are heterogeneous. Specific time estimates must be interpreted with caution since they also depend on contextual factors such as the topic of the SR or team characteristics. However, we think that the median or mean estimates as well as the ranges provide a good orientation.
5. Conclusion
Evidence on resource use during SR production is limited to studies reporting mostly on the resource “time” - often not under real life conditions. To be able to gain a more comprehensive understanding of the resource requirements for SRs, future studies need to assess resource use prospectively across various types of reviews (e.g. intervention, DTA, prognosis) and across different teams and settings to better reach generalizable estimates. Based on the identified evidence, the areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.
Funding source
This work was partly supported by funds from the EU funded COST Action EVBRES (CA17117) and internal funds from Danube University Krems.
None of the authors report any financial conflicts of interest with respect to the topic of this manuscript. All authors have a general interest in evidence synthesis methods. Some authors are associated with groups, conferences, and tools focusing on evidence synthesis methods: Barbara Nussbaumer-Streit and Gerald Gartlehner are co-convenors, Lisa Affengruber associate co-convenor of the Cochrane Rapid Reviews Methods Group. Raluca Sfetcu is a member of the JBI method group for "Systematic reviews of etiology and risk". Patricia Martinez is associated with the Techné Co-word research group that has patented knowledge engineering software that is used for mapping. Rene Spijker is on the organizing committee of ICASR, member of the Cochrane Information Specialist Executive, and Editorial Board Member of BMJ Evidence Based Medicine. James Thomas is on the organizing committee of ICASR (International Collaboration for the automation of systematic reviews); a co-senior scientific editor for the Cochrane Handbook; leads / contributes to research projects on systematic review automation; and leads the team that develops and supports EPPI-Reviewer, a software platform for evidence synthesis.
Acknowledgments
We would like to thank Sandra Hummel for administrative support throughout this project and Genevieve Iseult Eldredge for proofreading the manuscript. We would also like to thank Hans Lund and the members of the COST Action EVBRES (especially from working group 3) for their ideas and comments to this project.
Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.
Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].
Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
Agency for Healthcare Research and Quality (US),
Rockville (MD)2013
A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
Evidence Based Library and Information Practice.2019; 14: 80-83
00171-2&id=gr1.jpg){kind=link}
00171-2&id=gr2.jpg){kind=link}