Advertisement

Resource use during systematic review production varies widely: a scoping review

Open AccessPublished:June 03, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.05.019

      Highlights

      • Evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions.
      • Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time.
      • Lack of experience and domain knowledge, lack of collaborative and supportive software, as well as lack of good communication and management can increase resource use during the systematic review process.

      Abstract

      Objective

      We aimed to map the resource use during systematic review (SR) production and reasons why steps of the SR production are resource intensive to discover where the largest gain in improving efficiency might be possible.

      Study design and setting

      We conducted a scoping review. An information specialist searched multiple databases (e.g., Ovid MEDLINE, Scopus) and implemented citation-based and grey literature searching. We employed dual and independent screenings of records at the title/abstract and full-text levels and data extraction.

      Results

      We included 34 studies. Thirty-two reported on the resource use—mostly time; four described reasons why steps of the review process are resource intensive. Study selection, data extraction, and critical appraisal seem to be very resource intensive, while protocol development, literature search, or study retrieval take less time. Project management and administration required a large proportion of SR production time. Lack of experience, domain knowledge, use of collaborative and SR-tailored software, and good communication and management can be reasons why SR steps are resource intensive.

      Conclusion

      Resource use during SR production varies widely. Areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.

      Keywords

      Tabled 1
      Key findings:
      • Overall we identified 34 studies. Of these, 32 reported on resource use: three looked at resource use across the complete SR production process, while others only assessed the resource use of selected steps of the review process. Mostly, studies reported on the resource “time.”
      • Administration and project management, study selection, data extraction, and critical appraisal seem to be very resource intensive, varying with the number of included studies, while protocol development, literature search, and study retrieval take less time.
      • Four studies reported on reasons why steps are resource intensive: lack of experience and domain knowledge, lack of collaborative and SR-tailored software, doing steps manually instead of using supportive software, and lack of good communication and management of the SR process.
      What this adds to what is known: Our scoping review is the first to give an overview of the resource use during SR production. It shows that evidence on resource use is limited to studies reporting mostly on the resource “time” and not always under real life conditions. It also reveals that administration/project management is a time-consuming task that should be considered an important part of the SR process.
      What is the implication, what should change now: Based on our results, methods and tools to support project management and administration throughout a project, as well as methods and tools to speed up study selection, data extraction, and critical appraisal could help save resources.

      1. Introduction

      Well-conducted systematic reviews (SRs) are considered the most reliable form of evidence syntheses because they employ high methodological standards in summarizing primary research. SRs play an important role as support for evidence-based clinical and health policy decision-making. However, conducting a SR is very resource intensive and can take up to 2 years for completion [
      • Ganann R
      • Ciliska D
      • Thomas H.
      Expediting systematic reviews: methods and implications of rapid reviews.
      ,

      Hartling L, Guise JM, Kato E, Anderson J, Aronson N, Belinson S, et al. Agency for healthcare research and quality (US). 2015:02.

      ]. This often does not meet the needs of decision-makers, especially in times where evidence syntheses must answer urgent questions such as during the ongoing coronavirus pandemic.
      SRs are also essential in primary research. Systematically reviewing the existing evidence before starting a new study is important to ensure high quality and relevant primary research [
      • Clarke M
      • Hopewell S
      • Chalmers I.
      Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting.
      ]. Knowing all the studies in a field helps focus on topics and research questions that require new studies. In addition, learning from earlier studies helps optimally design new studies [
      • Robinson KA
      • Brunnhuber K
      • Ciliska D
      • Juhl CB
      • Christensen R
      • Lund H.
      Evidence-based research series-paper 1: what evidence-based research is and why is it important?.
      ,
      • Lund H
      • Juhl CB
      • Nørgaard B
      • Draborg E
      • Henriksen M
      • Andreasen J
      • et al.
      Evidence-based research series-paper 2: using an evidence-based research approach before a new study is conducted to ensure value.
      ]. Ideally, a comprehensive up-to-date SR informs every new study. However, as the methodological standards in SR production are very high, the steps to develop or update a SR are complex and resource intensive. This can keep primary researchers from conducting or updating a SR [
      • Clayton GL
      • Smith IL
      • Higgins JPT
      • Mihaylova B
      • Thorpe B
      • Cicero R
      • et al.
      The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials.
      ].
      According to Cochrane, a SR is “a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyses data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyses and summaries the results of the included studies” [

      Cochrane Community. Glossary [Available from: https://cdev.cochrane.org/glossary#letter-S.

      ]. To conduct a SR, certain steps must be completed, from formulating a focused research question to conducting a comprehensive search strategy, screening the identified literature, and critically appraising and synthesizing the primary studies [
      • Tsafnat G
      • Glasziou P
      • Choong MK
      • Dunn A
      • Galgani F
      • Coiera E.
      Systematic review automation technologies.
      ]. All these steps require resources such as time or money to complete. Performing a high-quality systematic search, for example, requires an information specialist's expertise and time. A recent study showed that information specialists needed an average aggregated time of 26.9 hours when developing a search strategy [
      • Bullers K
      • Howard AM
      • Hanson A
      • Kearns WD
      • Orriola JJ
      • Polo RL
      • et al.
      It takes longer than you think: librarian time spent on systematic review tasks.
      ]. Another study assessed the resource need for completing a SR with meta-analyses as ranging from 216 to 2518 hours, depending on the number of studies included and the comparisons and outcomes assessed [
      • Allen IE
      • Olkin I.
      Estimating time to conduct a meta-analysis from number of citations retrieved.
      ].
      The production and update of SRs must become more resource efficient to meet the time-sensitive needs of clinical and health policy decision-makers and to facilitate the uptake of an evidence-based research approach by primary researchers. To discover where the largest gain in improving efficiency might be possible, we wanted to map the resource use of different SR steps. This could help identify the most resource-intensive areas in the review process. In addition, we wanted to know the reasons why certain steps of the review process are resource intensive, in order to identify suitable methods to address them. Therefore, we conducted a scoping review of the published literature mapping the resource use of each step of the SR process and the reasons why steps of the SR process are resource intensive. Specifically, we strove to answer the following two key questions (KQs):
      • KQ 1. How many resources (e.g., time, costs) do different steps of the SR production consume?
      • KQ 2. What are the reasons why some steps of the SR production are resource intensive?

      2. Materials and methods

      This scoping review is part of working group 3’s work within the EVBRES (EVidence-Based RESearch) – COST Action CA17117, funded by the European Union (www.evbres.eu). The protocol for this scoping review was published a priori via the Open Science Framework (https://osf.io/fby54/) [

      Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.

      ].

      2.1 Study design

      We conducted a scoping review. Within EVBRES, we define a scoping review as a form of knowledge synthesis that addresses an exploratory research question aimed at mapping key concepts, theories, types and sources of evidence, and gaps in research related to a defined area or field by systematically searching, selecting, and synthesizing existing knowledge. We followed the guidance given by Arksey and O'Malley [
      • Arksey H
      • O'Malley L
      Scoping studies: towards a methodological framework.
      ], Levac et al., and Peters et al. [
      • Levac D
      • Colquhoun H
      • O'Brien KK
      Scoping studies: advancing the methodology.
      ,
      • Peters MDJ
      • Marnie C
      • Tricco AC
      • Pollock D
      • Munn Z
      • Alexander L
      • et al.
      Updated methodological guidance for the conduct of scoping reviews.
      ]. We reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Extension for Scoping Reviews (PRISMA-ScR) [
      • Tricco AC
      • Lillie E
      • Zarin W
      • O'Brien KK
      • Colquhoun H
      • Levac D
      • et al.
      PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.
      ].

      2.2 Information sources and search strategies

      The search for this scoping review followed the three-step process recommended by the Joanna Briggs Institute [

      Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). in: aromataris e, munn z (editors). jbi manual for evidence synthesis, JBI, 2020 2020 [Available from: https://wiki.jbi.global/display/MANUAL/Chapter+11%3A+Scoping+reviews.

      ]:
      • 1)
        In a first step, an information specialist (IK) conducted a focused search of Ovid MEDLINE and Current Contents Connect (Web of Science) in November 2019. We screened these results to identify relevant studies for inclusion. Using PubReMiner and AntConc [

        Koster JA. PubReMiner 2014 [Available from: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi.

        ,
        • Anthony L.
        AntConc (Version 3.5.9) [Computer Software].
        ], the included studies were analyzed to identify relevant text words contained in the title and abstract as well as Medical Subject Headings (MeSH) terms.
      • 2)
        Based on search terms derived from these included studies, IK developed a second, more comprehensive search strategy and searched the following databases in May 2020: Ovid MEDLINE, Scopus (Elsevier), Science Citation Index Expanded, Social Sciences Citation Index, and Current Contents Connect (all via Web of Science). The Ovid MEDLINE strategy was reviewed by a second information specialist (RS) in accordance with the Peer Review of Electronic Search Strategies (PRESS) guideline [
        • McGowan J
        • Sampson M
        • Salzwedel DM
        • Cogo E
        • Foerster V
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        ].
      • 3)
        Using the studies identified by these searches, we conducted citation-based searches: manual screening of reference lists, forward citation tracking (via Scopus in May 2020), and a similar articles search (via PubMed, limited to the first 50 linked references for each seed article, in May 2020).
      The database searches were limited to studies published since 2009 (the year when PRISMA was published). Last, to identify grey literature, we contacted experts in the field and screened the latest proceedings of the Cochrane Colloquium [

      The Cochrane Collaboration. Advances in evidence synthesis: special issue cochrane database of systematic reviews. 2020;(9 Suppl 1) 2020 [Available from: doi:10.1002/14651858.CD202001.

      ] to also include new studies that might not have been published yet. Details on the search strategies are available in web appendix 1.

      2.3 Eligibility criteria

      The eligibility criteria are specified in Table 1. For KQ 1, we included studies that assessed the resource needs of one or more steps of the review process. Studies that tracked resource use when conducting SRs and those that asked for resource use via a survey as well as studies that modeled resource use were eligible. If a study only reported relative measures of resource use, such as time or workload saved, without specifying the absolute resource use of the task of interest, the study was excluded. To define the SR production steps, we used the list of steps provided by Tsafnat et al. 2014 [
      • Tsafnat G
      • Glasziou P
      • Choong MK
      • Dunn A
      • Galgani F
      • Coiera E.
      Systematic review automation technologies.
      ] and added other important steps: “critical appraisal” and “grade the certainty of evidence,” [

      Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].

      ]. When mapping the results, we identified an additional SR production step and added “Administration/project management” a posteriori (Fig. 1). For KQ 2, we were interested in qualitative studies that asked SR authors via interviews or surveys why some steps of the SR production are resource intensive. We limited the publication date to 2009 because the current reporting standards, the PRISMA guidelines, were published at that time.
      Table 1Eligibility criteria.
      Inclusion criteriaExclusion criteria
      TopicKQ1: Studies reporting the resource use of producing or updating SRs
      resource use must already be mentioned in the abstract
      Studies not reporting resource use
      KQ2: Studies reporting reasons why steps of the review production are resource intensive
      ConceptKQ1 and KQ2: Addressing the resource use of one or more steps of a SR (as depicted in Fig. 1) of health interventions or diagnostic or prognostic studiesStudies assessing other steps such as the dissemination of a SR's results
      OutcomesKQ1:Any other outcomes, such as time or money saved, without specification of the total resource amount spent on a task
      • Time spent on tasks (e.g., minutes per task)
      • Effort of personnel (e.g., full-time equivalent)
      • Costs (e.g., salary, license fees)
      • Material used
      KQ2:
      • Perceived reasons why some steps are/ are not resource intensive (barriers/ facilitators)
      • Perceived resource-intensive steps
      Study design/ publication typeKQ1:Other publications (e.g., editorials, letters)
      • SRs
      • Empirical studies: measuring resource use by tracking
      • Surveys: asking about resource use
      • Simulation studies: modeling resource use
      KQ2:
      • Qualitative studies (e.g., interviews, open surveys)
      Full text availabilityIf full text was not retrievable via our libraries
      Timing2009–2020
      LanguageAll languages
      Abbreviations: KQ, key question, SR, systematic review
      a resource use must already be mentioned in the abstract
      Figure 1
      Figure 1Steps of a systematic review. *added a priori to Tsafnat et al.’s list
      [
      • Tsafnat G
      • Glasziou P
      • Choong MK
      • Dunn A
      • Galgani F
      • Coiera E.
      Systematic review automation technologies.
      ]
      . ** added after synthesis

      2.4 Selection of sources of evidence

      After piloting the abstract screening with 50 records, the author team used Covidence (www.covidence.org) to dually and independently screen records based on titles and abstracts and full texts. We resolved any screening discrepancies at the abstract or full-text level by discussion or by consultation with a third author. We stored records in the reference management software EndNote [

      Clarivate. EndNote X8.

      ].

      2.5 Data charting process and data items

      We developed and piloted a data extraction form using Google Forms. Two team members independently extracted data; a third author made final decisions in cases of discrepancies.

      2.6 Data synthesis

      We mapped the results of the scoping review in a summary table, grouping them along the steps of the review process. Because the goal of this scoping review was to descriptively map the resource use of SR production steps rather than to derive cause–effect relationships, we did not apply a formal certainty of evidence or risk of bias (RoB) assessment.

      3. Results

      We included 34 studies (32 quantitative primary studies, 1 qualitative primary study, 1 SR) that were published in 38 publications [
      • Bullers K
      • Howard AM
      • Hanson A
      • Kearns WD
      • Orriola JJ
      • Polo RL
      • et al.
      It takes longer than you think: librarian time spent on systematic review tasks.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Chang LK
      • Trikalinos TA.
      Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Trikalinos TA
      • Kong Win Chang L
      Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
      ,
      • Balk EM
      • Chung M
      • Hadar N
      • Patel K
      • Yu WW
      • Trikalinos TA
      • et al.
      AHRQ Methods for Effective Health Care. Accuracy of Data Extraction of Non-English Language Trials with Google Translate.
      ,
      • Bramer WM
      • Rethlefsen ML
      • Mast F
      • Kleijnen J.
      Evaluation of a new method for librarian-mediated literature searches for systematic reviews.
      ,
      • Chapman AL
      • Morgan LC
      • Gartlehner G.
      Semi-automating the manual literature search for systematic reviews increases efficiency.
      ,
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Clark JM
      • Sanders S
      • Carter M
      • Honeyman D
      • Cleo G
      • Auld Y
      • et al.
      Improving the translation of search strategies using the polyglot search translator: A randomized controlled trial.
      ,
      • Cooper C
      • Booth A
      • Britten N
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ,
      • Cooper C
      • Bou JT
      • Varley-Campbell J.
      Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example.
      ,
      A visual approach to validate the selection review of primary studies in systematic reviews: A replication study.
      ,
      • Felizardo KR
      • Salleh N
      • Martins RM
      • Mendes E
      • Macdonell SG
      • Maldonado JC
      Using visual text mining to support the study selection activity in systematic literature reviews.
      ,
      • Giummarra MJ
      • Lau G
      • Gabbe BJ.
      Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.
      ,
      • Grames EM
      • Stillman AN
      • Tingley MW
      • Elphick CS.
      An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks.
      ,

      Gresham G, Matsumura S, Li T. Faster may not be better: data abstraction for systematic reviews. Cochrane Colloquium; Hyderabad. AR I2014.

      ,
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ,
      • Hartling L
      • Bond K
      • Vandermeer B
      • Seida J
      • Dryden DM
      • Rowe BH.
      Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.
      ,
      • Hausner E
      • Guddat C
      • Hermanns T
      • Lampert U
      • Waffenschmidt S.
      Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
      ,
      • Hoang L
      • Schneider J.
      Opportunities for computer support for systematic reviewing - a gap analysis.
      ,
      • Horton J
      • Vandermeer B
      • Hartling L
      • Tjosvold L
      • Klassen TP
      • Buscemi N.
      Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
      ,
      • Jelicic Kadic A
      • Vucic K
      • Dosenovic S
      • Sapunar D
      • Puljak L
      Extracting data from figures with software was faster, with higher interrater reliability than manual extraction.
      ,
      • Jeyaraman MM
      • Rabbani R
      • Copstein L
      • Robson RC
      • Al-Yousif N
      • Pollock M
      • et al.
      Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.
      ,
      • Kim SY
      • Park JE
      • Lee YJ
      • Seo HJ
      • Sheen SS
      • Hahn S
      • et al.
      Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity.
      ,
      • Kwon Y
      • Lemieux M
      • McTavish J
      • Wathen N.
      Identifying and removing duplicate records from systematic review searches.
      ,
      • Li T
      • Saldanha IJ
      • Jap J
      • Smith BT
      • Canner J
      • Hutfless SM
      • et al.
      A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
      ,
      • Major MP
      • Warren S
      • Flores-Mir C.
      Survey of systematic review authors in dentistry: challenges in methodology and reporting.
      ,
      • Mathes T
      • Klasen P
      • Pieper D.
      Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.
      ,
      • Mortensen ML
      • Adam GP
      • Trikalinos TA
      • Kraska T
      • Wallace BC.
      An exploration of crowdsourcing citation screening for systematic reviews.
      ,
      • Nama N
      • Sampson M
      • Barrowman N
      • Sandarage R
      • Menon K
      • Macartney G
      • et al.
      Crowdsourcing the citation screening process for systematic reviews: validation study.
      ,
      • Petersen H
      • Poon J
      • Poon SK
      • Loy C.
      Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities.
      ,
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ,
      • Pradhan R
      • Hoaglin DC
      • Cornell M
      • Liu W
      • Wang V
      • Yu H.
      Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses.
      ,
      • Saleh AA
      • Ratajeski MA
      • Bertolet M.
      Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.
      ,
      • Shea BJ
      • Hamel C
      • Wells GA
      • Bouter LM
      • Kristjansson E
      • Grimshaw J
      • et al.
      AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
      ,
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ,
      • Wang Z
      • Asi N
      • Elraiyah TA
      • Abu Dabrh AM
      • Undavalli C
      • Glasziou P
      • et al.
      Dual computer monitors to increase efficiency of conducting systematic reviews.
      ,
      • Williamson PO.
      Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
      ,
      • Wright K
      • Golder S
      • Rodriguez-Lopez R.
      Citation searching: a systematic review case study of multiple risk behaviour interventions.
      ] (Fig. 2: PRISMA study flow). In the following sections, we first summarize the characteristics of the included studies. We then present the resource use for single steps of the review process as well as across all the steps combined. Finally, we summarize the reasons why the review authors perceive some steps of the SR process as resource intensive.

      3.1 Characteristics of the included studies

      Thirty-two studies contributed to KQ1 and reported on resource use, mostly on time spent [
      • Bullers K
      • Howard AM
      • Hanson A
      • Kearns WD
      • Orriola JJ
      • Polo RL
      • et al.
      It takes longer than you think: librarian time spent on systematic review tasks.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Chang LK
      • Trikalinos TA.
      Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Trikalinos TA
      • Kong Win Chang L
      Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
      ,
      • Balk EM
      • Chung M
      • Hadar N
      • Patel K
      • Yu WW
      • Trikalinos TA
      • et al.
      AHRQ Methods for Effective Health Care. Accuracy of Data Extraction of Non-English Language Trials with Google Translate.
      ,
      • Bramer WM
      • Rethlefsen ML
      • Mast F
      • Kleijnen J.
      Evaluation of a new method for librarian-mediated literature searches for systematic reviews.
      ,
      • Chapman AL
      • Morgan LC
      • Gartlehner G.
      Semi-automating the manual literature search for systematic reviews increases efficiency.
      ,
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Clark JM
      • Sanders S
      • Carter M
      • Honeyman D
      • Cleo G
      • Auld Y
      • et al.
      Improving the translation of search strategies using the polyglot search translator: A randomized controlled trial.
      ,
      • Cooper C
      • Booth A
      • Britten N
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ,
      • Cooper C
      • Bou JT
      • Varley-Campbell J.
      Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example.
      ,
      A visual approach to validate the selection review of primary studies in systematic reviews: A replication study.
      ,
      • Felizardo KR
      • Salleh N
      • Martins RM
      • Mendes E
      • Macdonell SG
      • Maldonado JC
      Using visual text mining to support the study selection activity in systematic literature reviews.
      ,
      • Giummarra MJ
      • Lau G
      • Gabbe BJ.
      Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.
      ,
      • Grames EM
      • Stillman AN
      • Tingley MW
      • Elphick CS.
      An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks.
      ,

      Gresham G, Matsumura S, Li T. Faster may not be better: data abstraction for systematic reviews. Cochrane Colloquium; Hyderabad. AR I2014.

      ,
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ,
      • Hartling L
      • Bond K
      • Vandermeer B
      • Seida J
      • Dryden DM
      • Rowe BH.
      Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.
      ,
      • Hausner E
      • Guddat C
      • Hermanns T
      • Lampert U
      • Waffenschmidt S.
      Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
      ,
      • Horton J
      • Vandermeer B
      • Hartling L
      • Tjosvold L
      • Klassen TP
      • Buscemi N.
      Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
      ,
      • Jelicic Kadic A
      • Vucic K
      • Dosenovic S
      • Sapunar D
      • Puljak L
      Extracting data from figures with software was faster, with higher interrater reliability than manual extraction.
      ,
      • Jeyaraman MM
      • Rabbani R
      • Copstein L
      • Robson RC
      • Al-Yousif N
      • Pollock M
      • et al.
      Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.
      ,
      • Kim SY
      • Park JE
      • Lee YJ
      • Seo HJ
      • Sheen SS
      • Hahn S
      • et al.
      Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity.
      ,
      • Kwon Y
      • Lemieux M
      • McTavish J
      • Wathen N.
      Identifying and removing duplicate records from systematic review searches.
      ,
      • Li T
      • Saldanha IJ
      • Jap J
      • Smith BT
      • Canner J
      • Hutfless SM
      • et al.
      A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
      ,
      • Mathes T
      • Klasen P
      • Pieper D.
      Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.
      ,
      • Mortensen ML
      • Adam GP
      • Trikalinos TA
      • Kraska T
      • Wallace BC.
      An exploration of crowdsourcing citation screening for systematic reviews.
      ,
      • Nama N
      • Sampson M
      • Barrowman N
      • Sandarage R
      • Menon K
      • Macartney G
      • et al.
      Crowdsourcing the citation screening process for systematic reviews: validation study.
      ,
      • Petersen H
      • Poon J
      • Poon SK
      • Loy C.
      Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities.
      ,
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ,
      • Pradhan R
      • Hoaglin DC
      • Cornell M
      • Liu W
      • Wang V
      • Yu H.
      Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses.
      ,
      • Saleh AA
      • Ratajeski MA
      • Bertolet M.
      Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.
      ,
      • Shea BJ
      • Hamel C
      • Wells GA
      • Bouter LM
      • Kristjansson E
      • Grimshaw J
      • et al.
      AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
      ,
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ,
      • Wang Z
      • Asi N
      • Elraiyah TA
      • Abu Dabrh AM
      • Undavalli C
      • Glasziou P
      • et al.
      Dual computer monitors to increase efficiency of conducting systematic reviews.
      ,
      • Williamson PO.
      Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
      ,
      • Wright K
      • Golder S
      • Rodriguez-Lopez R.
      Citation searching: a systematic review case study of multiple risk behaviour interventions.
      ]. Of these, three studies compared resource use across all steps of the review process [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ,
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ], while the other 29 focused on one or more single steps: study selection (title/abstract screening n = 7, obtaining full-text articles n = 1, full-text screening n = 4), literature search (n = 11), data extraction (n = 7), and critical appraisal (n = 4). Four studies helped answer KQ2 and described reasons why certain steps of the review process are resource intensive [
      • Balk EM
      • Chung M
      • Chen ML
      • Chang LK
      • Trikalinos TA.
      Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Trikalinos TA
      • Kong Win Chang L
      Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
      ,
      • Balk EM
      • Chung M
      • Hadar N
      • Patel K
      • Yu WW
      • Trikalinos TA
      • et al.
      AHRQ Methods for Effective Health Care. Accuracy of Data Extraction of Non-English Language Trials with Google Translate.
      ,
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Hoang L
      • Schneider J.
      Opportunities for computer support for systematic reviewing - a gap analysis.
      ,
      • Major MP
      • Warren S
      • Flores-Mir C.
      Survey of systematic review authors in dentistry: challenges in methodology and reporting.
      ]. In web appendix 2, we summarize the characteristics of all the included studies. In web appendix 3, we map the resource use to the steps of the SR process.

      3.2 Comparative resource use across all steps of the systematic review process

      Three studies assessed the resource use required to conduct complete SRs [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ,
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ]. Haddaway et al. [
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ] and Pham et al. [
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ] simulated SR production and estimated the resource needs for each step based on survey responses from SR authors or on the literature. According to Pham et al., the mean time needed to conduct all SR tasks was 881 person-hours, ranging from 243 to 1752 person-hours, depending on the scope of the SR. When comparing different steps of the review process, protocol development and literature search were less resource intensive than selecting studies and extracting data, which together took roughly 50 % of all the time resources needed [
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ]. Haddaway et al. simulated that the average time to conduct a SR is 1312 person-hours. They also concluded that study selection as well as data extraction and critical appraisal of studies are very time consuming. Searching, retrieving full texts, and assembling a library of relevant studies required less time than most stages. The largest proportion of time, however, was needed for administration and planning [
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ]. Clark et al. [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ] published the only empirical study tracking the time needed to conduct one SR with the help of automation tools. Their SR identified 1694 abstracts and finally included 8 publications. They completed it in 61 person-hours. The least time-consuming tasks were searches, deduplication, and obtaining full texts. The most time-consuming steps were administration and project management, abstract and full-text screening, data extraction, critical appraisal, and data synthesis [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ].

      3.3 Resource use during literature search

      The median time for the whole literature search process ranged from 7.85 to 22 hours [
      • Bullers K
      • Howard AM
      • Hanson A
      • Kearns WD
      • Orriola JJ
      • Polo RL
      • et al.
      It takes longer than you think: librarian time spent on systematic review tasks.
      ,
      • Saleh AA
      • Ratajeski MA
      • Bertolet M.
      Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.
      ]. Studies also reported the resource use for different literature search components. Developing a search strategy for a SR took 1 to 13.5 hours, depending on the software support (text mining) used [
      • Bullers K
      • Howard AM
      • Hanson A
      • Kearns WD
      • Orriola JJ
      • Polo RL
      • et al.
      It takes longer than you think: librarian time spent on systematic review tasks.
      ,
      • Bramer WM
      • Rethlefsen ML
      • Mast F
      • Kleijnen J.
      Evaluation of a new method for librarian-mediated literature searches for systematic reviews.
      ,
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Grames EM
      • Stillman AN
      • Tingley MW
      • Elphick CS.
      An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks.
      ,
      • Hausner E
      • Guddat C
      • Hermanns T
      • Lampert U
      • Waffenschmidt S.
      Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
      ]. Translating a search strategy for another database took 11 to 79 minutes manually and 6 to 57 minute per database using Polyglot Search Translator [
      • Clark JM
      • Sanders S
      • Carter M
      • Honeyman D
      • Cleo G
      • Auld Y
      • et al.
      Improving the translation of search strategies using the polyglot search translator: A randomized controlled trial.
      ]. Deduplication of records was estimated to take 3 to 10 minute per SR, depending on the software used; however, the authors also report deduplicates missed by the software that required manual reduplicating later in the process [
      • Kwon Y
      • Lemieux M
      • McTavish J
      • Wathen N.
      Identifying and removing duplicate records from systematic review searches.
      ].
      Studies reported the resource use of different additional search approaches: manual reference list checking of included studies can take 8 hours compared to 3 hours when using Scopus [
      • Chapman AL
      • Morgan LC
      • Gartlehner G.
      Semi-automating the manual literature search for systematic reviews increases efficiency.
      ]. Hand searching, the manual examination of content from relevant journals or conference proceedings, ranged from 6 to 60 minute per source [
      • Cooper C
      • Booth A
      • Britten N
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ]. Contacting authors took nearly 7 hours in a study that contacted 88 authors [
      • Cooper C
      • Bou JT
      • Varley-Campbell J.
      Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example.
      ]. Citation chasing required about 1 hour [
      • Cooper C
      • Booth A
      • Britten N
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ], and web searching added from 8 hours [
      • Cooper C
      • Booth A
      • Britten N
      • Garside R.
      A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
      ] to 3 days [
      • Wright K
      • Golder S
      • Rodriguez-Lopez R.
      Citation searching: a systematic review case study of multiple risk behaviour interventions.
      ]. The median time to search grey literature took 85 minutes (range: 20–3480) [
      • Saleh AA
      • Ratajeski MA
      • Bertolet M.
      Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.
      ].

      3.4 Resource use during study selection

      Team members were able to screen from 0.13 to 2.88 abstracts per minute [
      A visual approach to validate the selection review of primary studies in systematic reviews: A replication study.
      ,
      • Felizardo KR
      • Salleh N
      • Martins RM
      • Mendes E
      • Macdonell SG
      • Maldonado JC
      Using visual text mining to support the study selection activity in systematic literature reviews.
      ,
      • Giummarra MJ
      • Lau G
      • Gabbe BJ.
      Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.
      ,
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ]. Conflict resolution–often necessary in dual screening processes–took on average 5 minute per conflict, and retrieving full-text articles took 4 minutes per full text [
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ].
      Different approaches to title/abstract screening exist, and although they can reduce the time needed for this step, they may also increase the full-text screening burden. Shemilt et al. [
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ] analyzed the following approaches: (1) single screening; (2) single screening with semiautomated software (one person screens abstracts and automatically excludes those deemed irrelevant by the algorithm); (3) dual screening (two people independently screen all abstracts and discuss conflicting decisions); and (4) safety first (two people independently screen all abstracts, all potentially relevant abstracts are included for full-text screening). These approaches were all followed by dual independent full-text screening. Based on a project with 12477 abstracts, the semiautomated approach took the least time (572 hours) and cost the least (£ 37,860), while dual screening took twice as long (1089 hours) and cost nearly twice as much (£ 75139) [
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ].
      Two studies assessed a type of crowdsourcing for title/abstract screening. While experts cost between $ 3034 and $ 8777 to complete such a task, the crowd costs ranged between $ 458 and $ 2223. However, it required the crowd 4 to 17 days to complete the screening [
      • Mortensen ML
      • Adam GP
      • Trikalinos TA
      • Kraska T
      • Wallace BC.
      An exploration of crowdsourcing citation screening for systematic reviews.
      ]. In another study, the median time to acquire enough assessments per citation was 42 days [
      • Nama N
      • Sampson M
      • Barrowman N
      • Sandarage R
      • Menon K
      • Macartney G
      • et al.
      Crowdsourcing the citation screening process for systematic reviews: validation study.
      ]. For acquiring enough assessments for all full texts, the crowd took another median of 36 days [
      • Nama N
      • Sampson M
      • Barrowman N
      • Sandarage R
      • Menon K
      • Macartney G
      • et al.
      Crowdsourcing the citation screening process for systematic reviews: validation study.
      ].
      Full-text screening took people from 4.3 to 5 minutes per full text [
      • Giummarra MJ
      • Lau G
      • Gabbe BJ.
      Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.
      ,
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ]. Conflict resolution took 5 minutes per full text [
      • Shemilt I
      • Khan N
      • Park S
      • Thomas J.
      Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
      ].
      One study showed that diagnostic test accuracy (DTA) reviews have on average 185 % more workload during abstract screening and 167 % more during full-text screening, because searches for DTA reviews identify many more records [
      • Petersen H
      • Poon J
      • Poon SK
      • Loy C.
      Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities.
      ].

      3.5 Resource use during data extraction

      Extracting major information on study design, participants, and results took one person an average of 41 to 65 minute per study [

      Gresham G, Matsumura S, Li T. Faster may not be better: data abstraction for systematic reviews. Cochrane Colloquium; Hyderabad. AR I2014.

      ,
      • Wang Z
      • Asi N
      • Elraiyah TA
      • Abu Dabrh AM
      • Undavalli C
      • Glasziou P
      • et al.
      Dual computer monitors to increase efficiency of conducting systematic reviews.
      ]. Using two monitors instead of one helped reduce the time spent on data extraction [
      • Wang Z
      • Asi N
      • Elraiyah TA
      • Abu Dabrh AM
      • Undavalli C
      • Glasziou P
      • et al.
      Dual computer monitors to increase efficiency of conducting systematic reviews.
      ]; experience in data extraction was also associated with less time spent on this task [
      • Horton J
      • Vandermeer B
      • Hartling L
      • Tjosvold L
      • Klassen TP
      • Buscemi N.
      Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
      ]. While single data abstraction and verification took on average 107 minutes per study, doing dual independent data abstraction took 172 minutes [
      • Li T
      • Saldanha IJ
      • Jap J
      • Smith BT
      • Canner J
      • Hutfless SM
      • et al.
      A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
      ]. Data extraction from trial registry entries took on average 40 minutes per study [
      • Pradhan R
      • Hoaglin DC
      • Cornell M
      • Liu W
      • Wang V
      • Yu H.
      Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses.
      ], while manual data extraction took on average 11 to 13 minutes per figure; using software reduced the time spent to 5–6 minutes per figure [
      • Jelicic Kadic A
      • Vucic K
      • Dosenovic S
      • Sapunar D
      • Puljak L
      Extracting data from figures with software was faster, with higher interrater reliability than manual extraction.
      ].
      Sometimes the SR team must translate a study into English before being able to extract data. Using Google Translate took on average 15 to 60 minutes per study, depending on the publication's original language (Spanish: 15 minutes to Chinese: 60 minute) [
      • Balk EM
      • Chung M
      • Chen ML
      • Chang LK
      • Trikalinos TA.
      Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
      ,
      • Balk EM
      • Chung M
      • Chen ML
      • Trikalinos TA
      • Kong Win Chang L
      Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
      ].

      3.6 Resource use during critical appraisal

      On average, one person needed 21 minutes to apply the Cochrane RoB Tool 1.0 per RCT [
      • Hartling L
      • Bond K
      • Vandermeer B
      • Seida J
      • Dryden DM
      • Rowe BH.
      Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.
      ], 9.5 minutes to apply RoBANS, and 10.45 minutes for MINORS [
      • Kim SY
      • Park JE
      • Lee YJ
      • Seo HJ
      • Sheen SS
      • Hahn S
      • et al.
      Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity.
      ]. Applying ROBINS-I plus finding consensus took the longest, at 48.45 minutes, while applying ROB-NRSE and reaching consensus took on average 36.98 minutes [
      • Jeyaraman MM
      • Rabbani R
      • Copstein L
      • Robson RC
      • Al-Yousif N
      • Pollock M
      • et al.
      Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.
      ]. Authors needed 10 to 15 minutes to critically appraise a SR using the AMSTAR tool [
      • Shea BJ
      • Hamel C
      • Wells GA
      • Bouter LM
      • Kristjansson E
      • Grimshaw J
      • et al.
      AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
      ].

      3.7 Reasons for perceived resource intensity

      Facilitators: Methodological experience and content knowledge of team members, existing data extraction sheets from former projects, blocking time for the SR, daily project meetings to discuss upcoming questions, and writing the protocol in past tense all contributed to speeding up the process. In addition, team members’ physical proximity allowed for ongoing communication and short time lapses between tasks, and familiarity with the used software tools helped [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ].
      Barriers: Lack of domain expertise, juggling other projects with competing deadlines, noisy surroundings, resource unavailability, poor internet, and software incompatibilities and limitations (e.g., automation of only one task) increased the resource use [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ]. Lack of collaborative platforms, of literature search experience [
      • Major MP
      • Warren S
      • Flores-Mir C.
      Survey of systematic review authors in dentistry: challenges in methodology and reporting.
      ], and manually doing tasks such as abstract screening acted as further barriers [
      • Hoang L
      • Schneider J.
      Opportunities for computer support for systematic reviewing - a gap analysis.
      ]. Hoang et al. also identified a gap between the available SR support tools and the tools used. The majority of review authors still rely only on Excel and EndNote, not using other SR-tailored tools. Potential explanations were that the reviewers are not aware of the other tools, might have limited access due to costs, or feel more comfortable using well-known tools [
      • Hoang L
      • Schneider J.
      Opportunities for computer support for systematic reviewing - a gap analysis.
      ]. Balk et al. highlighted that automated translation needs more resources for non-European languages and tables than for European languages, and if manuscripts are not available in Word or PDF formats [
      • Balk EM
      • Chung M
      • Chen ML
      • Chang LK
      • Trikalinos TA.
      Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
      ].

      4. Discussion

      To the best of our knowledge, this is the first scoping review mapping the resource use required to conduct a SR. Across all SR production steps, study selection, data extraction, and critical appraisal seem to be very resource intensive while protocol development, literature search, and study retrieval take less time [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Haddaway NR
      • Westgate MJ.
      Predicting the time needed for environmental systematic reviews and systematic maps.
      ,
      • Pham B
      • Bagheri E
      • Rios P
      • Pourmasoumi A
      • Robson RC
      • Hwee J
      • et al.
      Improving the conduct of systematic reviews: a process mining perspective.
      ]. Project management and coordination needed the largest proportion of SR production time. This is relevant for future initiatives that aim to make the SR production process more efficient, since this task is usually not in focus when thinking about SR production steps.
      We did not identify any study reporting on how much resources the certainty of evidence assessments requires, and only 6 studies reported resources other than time. We also did not identify a study focusing specifically on updating an existing SR; however, the resource use for individual steps of the SR process is probably similar when updating a review. The only study reporting resource use for the specific steps of formulating the review question, searching for existing SRs, writing a protocol, synthesis/ meta-analysis, and writing up the report was that of Clark et al. [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ]. This study was a case study of a single SR, so the generalizability is very limited.
      Our scoping review showed that literature search requires only a small proportion of the overall time required to complete a SR. This contradicts survey results among scientists in dentistry who perceived literature search as a particularly time-consuming challenge [
      • Major MP
      • Warren S
      • Flores-Mir C.
      Survey of systematic review authors in dentistry: challenges in methodology and reporting.
      ]. In the included studies of our scoping review, information specialists performed the search steps, which might explain why literature search did not take so much time though it is a very complex step in the SR process.
      The time needed for study selection, data extraction, and critical appraisal varied largely depending on the number of identified records and full texts. Other factors such as lack of experience or domain expertise might also increase resource use. In addition, the lack of using collaborative and supportive software increased the resource need [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ,
      • Hoang L
      • Schneider J.
      Opportunities for computer support for systematic reviewing - a gap analysis.
      ]. Although many supportive tools are available, the use of automation tools is still not very common in the SR community [
      • van Altena AJ
      • Spijker R
      • Olabarriaga SD.
      Usage of automation tools in systematic reviews.
      ]. Clark 2020a highlighted that the least time-consuming SR production tasks were generally those where the most automation tools were available, and vice versa [
      • Clark J
      • Glasziou P
      • Del Mar C
      • Bannach-Brown A
      • Stehlik P
      • Scott AM.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ].

      4.1 Limitations

      Our scoping review has several limitations. First, we focused only on the resource use of different steps and approaches, not on the validity and accuracy of the methods or tools. We plan to answer this in a follow-up scoping review (https://osf.io/9423z). Second, we did not assess the RoB of the included studies. Instead, we described the methodological approach of the included studies for transparency. Third, we limited the inclusion criteria to studies published from 2009 onward—the year when PRISMA was published. We might have missed older but informative and relevant studies. However, we think that the included studies provide a good overview of current resource use in SR production. Fourth, we included only studies that mention any type of resource use in the abstract. We considered this necessary to achieve a balance of sensitivity and specificity during the screening process. Fifth, the included studies are heterogeneous. Specific time estimates must be interpreted with caution since they also depend on contextual factors such as the topic of the SR or team characteristics. However, we think that the median or mean estimates as well as the ranges provide a good orientation.

      5. Conclusion

      Evidence on resource use during SR production is limited to studies reporting mostly on the resource “time” - often not under real life conditions. To be able to gain a more comprehensive understanding of the resource requirements for SRs, future studies need to assess resource use prospectively across various types of reviews (e.g. intervention, DTA, prognosis) and across different teams and settings to better reach generalizable estimates. Based on the identified evidence, the areas with the largest resource use are administration and project management, study selection, data extraction, and critical appraisal of studies.

      Funding source

      This work was partly supported by funds from the EU funded COST Action EVBRES ( CA17117 ) and internal funds from Danube University Krems .

      Author contribution

      Nussbaumer-Streit B: conceptualization, methodology, validation, formal analysis, investigation, data curation, writing – original draft, project administration. Ellen M: conceptualization, methodology, investigation, writing- review & editing. Klerings: conceptualization, literature search, writing-review & editing. Sfetcu R: conceptualization, methodology, investigation, writing- review & editing. Riva N: E: conceptualization, methodology, investigation, writing- review & editing. Mahmić-Kaknjo M: conceptualization, methodology, investigation, writing- review & editing. Poulentzas G: investigation, writing- review & editing. Martinez P: E: conceptualization, methodology, investigation, writing- review & editing. Baladia E: conceptualization, methodology, investigation, writing- review & editing. Ziganshina LE: E: conceptualization, methodology, investigation, writing- review & editing. Marqués ME: E: conceptualization, methodology, investigation, writing- review & editing. Aguilar L: E: conceptualization, methodology, investigation, writing- review & editing. Kassianos AP: E: conceptualization, methodology, investigation, writing- review & editing. Frampton G: conceptualization, methodology, investigation, writing- review & editing. Silva AG: E: conceptualization, methodology, investigation, writing- review & editing. Affengruber L: E: conceptualization, methodology, investigation, writing- review & editing. Spjker R: conceptualization, methodology, writing- review & editing. Thomas J: conceptualization, methodology, writing- review & editing. Berg RC: conceptualization, methodology, writing- review & editing. Kontogiani M: conceptualization, methodology, investigation, writing- review & editing. Sousa M: conceptualization, methodology, writing- review & editing. Kontogiorgis C: investigation, writing- review & editing. Gartlehner G: conceptualization, methodology, writing- review & editing.

      Conflict of interest

      None of the authors report any financial conflicts of interest with respect to the topic of this manuscript. All authors have a general interest in evidence synthesis methods. Some authors are associated with groups, conferences, and tools focusing on evidence synthesis methods: Barbara Nussbaumer-Streit and Gerald Gartlehner are co-convenors, Lisa Affengruber associate co-convenor of the Cochrane Rapid Reviews Methods Group. Raluca Sfetcu is a member of the JBI method group for "Systematic reviews of etiology and risk". Patricia Martinez is associated with the Techné Co-word research group that has patented knowledge engineering software that is used for mapping. Rene Spijker is on the organizing committee of ICASR, member of the Cochrane Information Specialist Executive, and Editorial Board Member of BMJ Evidence Based Medicine. James Thomas is on the organizing committee of ICASR (International Collaboration for the automation of systematic reviews); a co-senior scientific editor for the Cochrane Handbook; leads / contributes to research projects on systematic review automation; and leads the team that develops and supports EPPI-Reviewer, a software platform for evidence synthesis.

      Acknowledgments

      We would like to thank Sandra Hummel for administrative support throughout this project and Genevieve Iseult Eldredge for proofreading the manuscript. We would also like to thank Hans Lund and the members of the COST Action EVBRES (especially from working group 3) for their ideas and comments to this project.

      References

        • Ganann R
        • Ciliska D
        • Thomas H.
        Expediting systematic reviews: methods and implications of rapid reviews.
        Implement Sci. 2010; 5: 56
      1. Hartling L, Guise JM, Kato E, Anderson J, Aronson N, Belinson S, et al. Agency for healthcare research and quality (US). 2015:02.

        • Clarke M
        • Hopewell S
        • Chalmers I.
        Clinical trials should begin and end with systematic reviews of relevant evidence: 12 years and waiting.
        The Lancet. 2010; 376: 20-21
        • Robinson KA
        • Brunnhuber K
        • Ciliska D
        • Juhl CB
        • Christensen R
        • Lund H.
        Evidence-based research series-paper 1: what evidence-based research is and why is it important?.
        J Clin Epidemiol. 2021; 129: 151-157
        • Lund H
        • Juhl CB
        • Nørgaard B
        • Draborg E
        • Henriksen M
        • Andreasen J
        • et al.
        Evidence-based research series-paper 2: using an evidence-based research approach before a new study is conducted to ensure value.
        J Clin Epidemiol. 2021; 129: 158-166
        • Clayton GL
        • Smith IL
        • Higgins JPT
        • Mihaylova B
        • Thorpe B
        • Cicero R
        • et al.
        The INVEST project: investigating the use of evidence synthesis in the design and analysis of clinical trials.
        Trials. 2017; 18: 219
      2. Cochrane Community. Glossary [Available from: https://cdev.cochrane.org/glossary#letter-S.

        • Tsafnat G
        • Glasziou P
        • Choong MK
        • Dunn A
        • Galgani F
        • Coiera E.
        Systematic review automation technologies.
        Syst. 2014; 3: 74
        • Bullers K
        • Howard AM
        • Hanson A
        • Kearns WD
        • Orriola JJ
        • Polo RL
        • et al.
        It takes longer than you think: librarian time spent on systematic review tasks.
        J Med Libr Assoc. 2018; 106: 198-207
        • Allen IE
        • Olkin I.
        Estimating time to conduct a meta-analysis from number of citations retrieved.
        Jama. 1999; 282: 634-635
      3. Nussbaumer-Streit B, Ellen M, Klerings I, Gartlehner G, Thomas J, Mikkelsen LR, et al. Identifying resource intensive areas of systematic review production and updating – a scoping review 2020 [Available from: https://osf.io/8an4j.

        • Arksey H
        • O'Malley L
        Scoping studies: towards a methodological framework.
        Int J Soc Res Methodol. 2005; 8: 19-32
        • Levac D
        • Colquhoun H
        • O'Brien KK
        Scoping studies: advancing the methodology.
        Implement Sci. 2010; 5: 69
        • Peters MDJ
        • Marnie C
        • Tricco AC
        • Pollock D
        • Munn Z
        • Alexander L
        • et al.
        Updated methodological guidance for the conduct of scoping reviews.
        JBI Evid Synth. 2020; 18: 2119-2126
        • Tricco AC
        • Lillie E
        • Zarin W
        • O'Brien KK
        • Colquhoun H
        • Levac D
        • et al.
        PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation.
        Ann Intern Med. 2018; 169: 467-473
      4. Peters MDJ, Godfrey C, McInerney P, Munn Z, Tricco AC, Khalil H. Chapter 11: scoping reviews (2020 version). in: aromataris e, munn z (editors). jbi manual for evidence synthesis, JBI, 2020 2020 [Available from: https://wiki.jbi.global/display/MANUAL/Chapter+11%3A+Scoping+reviews.

      5. Koster JA. PubReMiner 2014 [Available from: https://hgserver2.amc.nl/cgi-bin/miner/miner2.cgi.

        • Anthony L.
        AntConc (Version 3.5.9) [Computer Software].
        Waseda University, Tokyo, Japan2020
        • McGowan J
        • Sampson M
        • Salzwedel DM
        • Cogo E
        • Foerster V
        • Lefebvre C.
        PRESS peer review of electronic search strategies: 2015 guideline statement.
        J Clin Epidemiol. 2016; 75: 40-46
      6. The Cochrane Collaboration. Advances in evidence synthesis: special issue cochrane database of systematic reviews. 2020;(9 Suppl 1) 2020 [Available from: doi:10.1002/14651858.CD202001.

      7. Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al. Cochrane handbook for systematic reviews of interventions version 6.2 (updated february 2021): Cochrane. 2021 [Available from: www.training.cochrane.org/handbook ].

      8. Clarivate. EndNote X8.

        • Balk EM
        • Chung M
        • Chen ML
        • Chang LK
        • Trikalinos TA.
        Data extraction from machine-translated versus original language randomized trial reports: a comparative study.
        Syst. 2013; 2: 97
        • Balk EM
        • Chung M
        • Chen ML
        • Trikalinos TA
        • Kong Win Chang L
        Assessing the Accuracy of Google Translate to Allow Data Extraction From Trials Published in Non-English Languages. AHRQ Methods for Effective Health Care.
        Agency for Healthcare Research and Quality (US), Rockville (MD)2013
        • Balk EM
        • Chung M
        • Hadar N
        • Patel K
        • Yu WW
        • Trikalinos TA
        • et al.
        AHRQ Methods for Effective Health Care. Accuracy of Data Extraction of Non-English Language Trials with Google Translate.
        Agency for Healthcare Research and Quality (US), Rockville (MD)2012
        • Bramer WM
        • Rethlefsen ML
        • Mast F
        • Kleijnen J.
        Evaluation of a new method for librarian-mediated literature searches for systematic reviews.
        Res. 2018; 9: 510-520
        • Chapman AL
        • Morgan LC
        • Gartlehner G.
        Semi-automating the manual literature search for systematic reviews increases efficiency.
        Health Info Libr J. 2010; 27: 22-27
        • Clark J
        • Glasziou P
        • Del Mar C
        • Bannach-Brown A
        • Stehlik P
        • Scott AM.
        A full systematic review was completed in 2 weeks using automation tools: a case study.
        J Clin Epidemiol. 2020; 121: 81-90
        • Clark JM
        • Sanders S
        • Carter M
        • Honeyman D
        • Cleo G
        • Auld Y
        • et al.
        Improving the translation of search strategies using the polyglot search translator: A randomized controlled trial.
        J Med Libr Assoc. 2020; 108: 195-207
        • Cooper C
        • Booth A
        • Britten N
        • Garside R.
        A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review.
        Syst. 2017; 6: 234
        • Cooper C
        • Bou JT
        • Varley-Campbell J.
        Evaluating the effectiveness, efficiency, cost and value of contacting study authors in a systematic review: a case study and worked example.
        BMC Med Res Methodol. 2019; 19: 45
      9. A visual approach to validate the selection review of primary studies in systematic reviews: A replication study.
        in: Felizardo KR Barbosa EF Maldonado JC Proceedings of the International Conference on Software Engineering and Knowledge Engineering. SEKE, 2013
        • Felizardo KR
        • Salleh N
        • Martins RM
        • Mendes E
        • Macdonell SG
        • Maldonado JC
        Using visual text mining to support the study selection activity in systematic literature reviews.
        International Symposium on Empirical Software Engineering and Measurement;. 2011;
        • Giummarra MJ
        • Lau G
        • Gabbe BJ.
        Evaluation of text mining to reduce screening workload for injury-focused systematic reviews.
        Inj Prev. 2019; 26: 26
        • Grames EM
        • Stillman AN
        • Tingley MW
        • Elphick CS.
        An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks.
        Methods Ecol Evol. 2019; (NIL_1-NIL_10)
      10. Gresham G, Matsumura S, Li T. Faster may not be better: data abstraction for systematic reviews. Cochrane Colloquium; Hyderabad. AR I2014.

        • Haddaway NR
        • Westgate MJ.
        Predicting the time needed for environmental systematic reviews and systematic maps.
        Conserv. 2019; 33: 434-443
        • Hartling L
        • Bond K
        • Vandermeer B
        • Seida J
        • Dryden DM
        • Rowe BH.
        Applying the risk of bias tool in a systematic review of combination long-acting beta-agonists and inhaled corticosteroids for persistent asthma.
        PLoS ONE. 2011; 6: 6
        • Hausner E
        • Guddat C
        • Hermanns T
        • Lampert U
        • Waffenschmidt S.
        Development of search strategies for systematic reviews: validation showed the noninferiority of the objective approach.
        J Clin Epidemiol. 2015; 68: 191-199
        • Hoang L
        • Schneider J.
        Opportunities for computer support for systematic reviewing - a gap analysis.
        Transform Digit Worlds. 2018; 10766 (2018): 367-377
        • Horton J
        • Vandermeer B
        • Hartling L
        • Tjosvold L
        • Klassen TP
        • Buscemi N.
        Systematic review data extraction: cross-sectional study showed that experience did not increase accuracy.
        J Clin Epidemiol. 2010; 63: 289-298
        • Jelicic Kadic A
        • Vucic K
        • Dosenovic S
        • Sapunar D
        • Puljak L
        Extracting data from figures with software was faster, with higher interrater reliability than manual extraction.
        J Clin Epidemiol. 2016; 74: 119-123
        • Jeyaraman MM
        • Rabbani R
        • Copstein L
        • Robson RC
        • Al-Yousif N
        • Pollock M
        • et al.
        Methodologically rigorous risk of bias tools for nonrandomized studies had low reliability and high evaluator burden.
        J Clin Epidemiol. 2020; 128: 140-147
        • Kim SY
        • Park JE
        • Lee YJ
        • Seo HJ
        • Sheen SS
        • Hahn S
        • et al.
        Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity.
        J Clin Epidemiol. 2013; 66: 408-414
        • Kwon Y
        • Lemieux M
        • McTavish J
        • Wathen N.
        Identifying and removing duplicate records from systematic review searches.
        J Med Libr Assoc. 2015; 103: 184-188
        • Li T
        • Saldanha IJ
        • Jap J
        • Smith BT
        • Canner J
        • Hutfless SM
        • et al.
        A randomized trial provided new evidence on the accuracy and efficiency of traditional vs. electronically annotated abstraction approaches in systematic reviews.
        J Clin Epidemiol. 2019; 115: 77-89
        • Major MP
        • Warren S
        • Flores-Mir C.
        Survey of systematic review authors in dentistry: challenges in methodology and reporting.
        J Dent Educ. 2009; 73: 471-482
        • Mathes T
        • Klasen P
        • Pieper D.
        Frequency of data extraction errors and methods to increase data extraction quality: a methodological review.
        BMC Med Res Methodol. 2017; 17: 152
        • Mortensen ML
        • Adam GP
        • Trikalinos TA
        • Kraska T
        • Wallace BC.
        An exploration of crowdsourcing citation screening for systematic reviews.
        Res. 2017; 8: 366-386
        • Nama N
        • Sampson M
        • Barrowman N
        • Sandarage R
        • Menon K
        • Macartney G
        • et al.
        Crowdsourcing the citation screening process for systematic reviews: validation study.
        J Med Internet Res. 2019; 21: e12953
        • Petersen H
        • Poon J
        • Poon SK
        • Loy C.
        Increased workload for systematic review literature searches of diagnostic tests compared with treatments: challenges and opportunities.
        JMIR Med Inform. 2014; 2: e11
        • Pham B
        • Bagheri E
        • Rios P
        • Pourmasoumi A
        • Robson RC
        • Hwee J
        • et al.
        Improving the conduct of systematic reviews: a process mining perspective.
        J Clin Epidemiol. 2018; 103: 101-111
        • Pradhan R
        • Hoaglin DC
        • Cornell M
        • Liu W
        • Wang V
        • Yu H.
        Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses.
        J Clin Epidemiol. 2019; 105: 92-100
        • Saleh AA
        • Ratajeski MA
        • Bertolet M.
        Grey literature searching for health sciences systematic reviews: a prospective study of time spent and resources utilized.
        Evid Based Libr Inf Pract. 2014; 9: 28-50
        • Shea BJ
        • Hamel C
        • Wells GA
        • Bouter LM
        • Kristjansson E
        • Grimshaw J
        • et al.
        AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
        J Clin Epidemiol. 2009; 62: 1013-1020
        • Shemilt I
        • Khan N
        • Park S
        • Thomas J.
        Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.
        Syst. 2016; 5: 140
        • Wang Z
        • Asi N
        • Elraiyah TA
        • Abu Dabrh AM
        • Undavalli C
        • Glasziou P
        • et al.
        Dual computer monitors to increase efficiency of conducting systematic reviews.
        J Clin Epidemiol. 2014; 67: 1353-1357
        • Williamson PO.
        Librarians' reported systematic review completion time ranges between 2 and 219 total hours with most variance due to information processing and instruction.
        Evidence Based Library and Information Practice. 2019; 14: 80-83
        • Wright K
        • Golder S
        • Rodriguez-Lopez R.
        Citation searching: a systematic review case study of multiple risk behaviour interventions.
        BMC Med Res Methodol. 2014; 14: 73
        • van Altena AJ
        • Spijker R
        • Olabarriaga SD.
        Usage of automation tools in systematic reviews.
        Res. 2019; 10: 72-82