Rapid review methods more challenging during COVID-19: commentary with a focus on 8 knowledge synthesis steps

  • Andrea C. Tricco
    Correspondence
    Corresponding author. Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada. Tel.: +1 416 864 6060x77521; fax: +1 416 864 6057.
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada

    Epidemiology Division, Dalla Lana School of Public Health and Institute for Health, Management, and Evaluation, University of Toronto, 6th Floor, 155 College Street, Toronto, Ontario M5T 3M7, Canada

    Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, School of Nursing, Queen's University, 92 Barrie Street, Kingston, Ontario K7L 3N6, Canada
    Search for articles by this author
  • Chantelle M. Garritty
    Affiliations
    Knowledge Synthesis Group, Ottawa Hospital Research Institute, Co-Convenor, Cochrane Rapid Reviews Methods Group, 1053 Carling Avenue, Ottawa, Ontario K1Y 4E9, Canada
    Search for articles by this author
  • Leah Boulos
    Affiliations
    Maritime SPOR SUPPORT Unit, 5790 University Avenue, Halifax, Nova Scotia B3H 1V7, Canada
    Search for articles by this author
  • Craig Lockwood
    Affiliations
    JBI, School of Public Health, The University of Adelaide, Level 3/55 King William Street, Adelaide, Australia

    Queens University, School of Nursing, 99 University Avenue, Kingston, Ontario K7L 3N6, Canada
    Search for articles by this author
  • Michael Wilson
    Affiliations
    McMaster Health Forum, McMaster University, 1280 Main Street West, MML-417, Hamilton, Ontario L8S 4L6, Canada

    Department of Health Evidence and Impact, McMaster University Medical Centre, 1280 Main St. West, Hamilton, Ontario L8S 4K1, Canada

    Centre for Health Economics and Policy Analysis, McMaster University, CRL Building 282, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada
    Search for articles by this author
  • Jessie McGowan
    Affiliations
    School of Epidemiology and Public Health, University of Ottawa, Alta Vista Campus, 101-600 Peter Morand Crescent, Ottawa, Ontario K1G5Z3, Canada
    Search for articles by this author
  • Michael McCaul
    Affiliations
    Centre for Evidence-based Health Care, Division of Epidemiology and Biostatistics, Department of Global Health, Stellenbosch University, Stellenbosch Central, Stellenbosch, South Africa
    Search for articles by this author
  • Brian Hutton
    Affiliations
    Ottawa Hospital Research Institute; University of Ottawa School of Epidemiology and Public Health, 1053 Carling Avenue, Ottawa, Ontario K1Y 4E9, Canada
    Search for articles by this author
  • Fiona Clement
    Affiliations
    Department of Community Health Sciences, Cumming School of Medicine, University of Calgary, 3rd Floor, TRW Building, 3280 Hospital Dr NW, Calgary, Alberta T2N 4N6, Canada
    Search for articles by this author
  • Nicole Mittmann
    Affiliations
    Canadian Agency for Drugs and Technologies in Health, 154 University Avenue, Suite 300, Toronto, Ontario M5H 3Y9, Canada

    Sunnybrook Research Institute, Sunnybrook Health Sciences Centre, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5, Canada

    Department of Pharmacology and Toxicology, Institute for Health Policy, Management and Evaluation, University of Toronto, 155 College St 4th Floor, Toronto, Ontario M5T 3M6, Canada
    Search for articles by this author
  • Declan Devane
    Affiliations
    School of Nursing and Midwifery, National University of Ireland Galway, Evidence Synthesis Ireland, 26 Upper Newcastle, Galway, H91 E3YV Ireland
    Search for articles by this author
  • Etienne V. Langlois
    Affiliations
    Partnership for Maternal, Newborn & Child Health (PMNCH), World Health Organization (WHO), 20 Avenue Appia, 1211 Geneva, Switzerland
    Search for articles by this author
  • Ahmed M. Abou-Setta
    Affiliations
    George & Fay Yee Centre for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, 367-753 McDermot Avenue, Winnipeg, Manitoba R3E 0T6, Canada
    Search for articles by this author
  • Catherine Houghton
    Affiliations
    School of Nursing and Midwifery, National University of Ireland Galway, 26 Upper Newcastle, Galway, H91 E3YV Ireland
    Search for articles by this author
  • Claire Glenton
    Affiliations
    Cochrane EPOC, Norwegian Institute of Public Health, PO Box 4404, Nydalen, N-0403 Oslo, Norway
    Search for articles by this author
  • Shannon E. Kelly
    Affiliations
    Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, 40 Ruskin Street, Ottawa, Ontario K1Y 4W7, Canada

    School of Epidemiology and Public Health, University of Ottawa, Alta Vista Campus, 101-600 Peter Morand Crescent, Ottawa, Ontario K1G5Z3, Canada
    Search for articles by this author
  • Vivian A. Welch
    Affiliations
    Bruyère Research Institute, 85 Primrose Avenue, Ottawa, Ontario K1R 6M1, Canada

    School of Epidemiology and Public Health, University of Ottawa, 75 Laurier Avenue E, Ottawa, Ontario K1N 6N5, Canada
    Search for articles by this author
  • Annie LeBlanc
    Affiliations
    VITAM Research Center in Sustainable Health, CIUSSS de la Capitale-Nationale, Department of Family and Emergency Medicine, Faculty of Medicine, Laval University, Ferdinand Vandry Pavillon, 1050 Avenue de la Médecine, Québec G1V 0A6, Canada
    Search for articles by this author
  • George A. Wells
    Affiliations
    Cardiovascular Research Methods Centre, University of Ottawa Heart Institute, 40 Ruskin Street, Ottawa, Ontario K1Y 4W7, Canada

    School of Epidemiology and Public Health, University of Ottawa, Alta Vista Campus, 101-600 Peter Morand Crescent, Ottawa, Ontario K1G5Z3, Canada
    Search for articles by this author
  • Ba’ Pham
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Simon Lewin
    Affiliations
    Cochrane EPOC, Norwegian Institute of Public Health, Oslo, Norway and Health Systems Research Unit, South African Medical Research Council, Cape Town, South Africa
    Search for articles by this author
  • Sharon E. Straus
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada

    Department of Geriatric Medicine, University of Toronto, 27 King's College Circle, Toronto, Ontario M5S 1A1, Canada
    Search for articles by this author
      What is new?

         Key findings

      • Guidance is available on the conduct of rapid reviews. However, the COVID-19 pandemic has created several unique challenges.
      • Challenges to the conduct of rapid reviews include the urgency of the request from decision-maker organizations, identification of and access to sources of evidence for inclusion in the rapid reviews, extrapolation of results from indirect evidence, and dissemination of results widely.

         What this adds to what is known?

      • There is a need for coordination of efforts internationally to reduce the risk of duplication, and to effectively use global collective evidence synthesis resources.
      • We outline several methodological challenges to the conduct of rapid reviews that have become apparent during the COVID-19 pandemic using an 8-step framework that follows the knowledge synthesis process.

         What is the implication and what should change now?

      • We offer several suggestions to help address the methodological challenges encountered during the conduct of rapid reviews on COVID-19, as well as future research.

      1. Introduction

      COVID-19 has driven the need for timely evidence to inform decision-making. Rapid evidence products can be particularly helpful for decision-makers (e.g., citizens, patients, health care providers, policy-makers) during COVID-19. The main types of rapid evidence products include inventories, rapid response briefs, and rapid reviews [
      • Hartling L.
      • Guise J.M.
      • Kato E.
      • Anderson J.
      • Aronson N.
      • Belinson S.
      • et al.
      EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews.
      ]. In this article, we focus on rapid reviews, which are “a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner [
      Cochrane's work on Rapid Reviews in response to COVID-19. Cochrane.
      ].” In a rapid review, several mechanisms are used to streamline the methods, such as narrowing the scope of the topic, parallelization of tasks (e.g., conducting screening and data abstraction simultaneously), using review short cuts (e.g., one team member screens citations from the literature search versus two), and using automation (e.g., using computer software to rank the literature search results in terms of relevance) [
      ].
      Guidance for the conduct of rapid reviews for health policy and systems research has been developed in collaboration with the World Health Organization (WHO) [
      ,
      • Langlois E.V.
      • Straus S.E.
      • Antony J.
      • King V.J.
      • Tricco A.C.
      Using rapid reviews to strengthen health policy and systems and progress towards universal health coverage.
      ]. The WHO guide recommends that researchers thoughtfully tailor their methods to the needs of decision-makers. The guide notes that there are several ways that rapid reviews can be streamlined to accommodate decision-makers’ needs related to both the scope of the review and timeliness across all steps of the review process.
      In this article, we outline several challenges based on our collective experience conducting COVID-19 rapid reviews predominantly for health care provider and policy decision-makers. The types of rapid reviews that the authors have conducted vary in scope including public health measures, clinical management, health-systems arrangements, and economic and social responses [
      COVID-END: COVID-19 evidence network to support decision-making. McMaster University.
      ], providing a wide range of experiences to draw on. We specifically describe issues we have encountered throughout the rapid review process (Fig. 1). The collective lessons we have learned through our collective experiences are also explored, as well as suggestions for future research.
      Figure thumbnail gr1
      Fig. 1This figure provides specific steps related to the rapid review process.

      2. Discussion

      2.1 Conceptualizing the question and scope through stakeholder involvement

      2.1.1 Challenges

      The conceptualization and scope of rapid reviews has been challenging during COVID-19. It is essential for researchers to involve decision-makers in the rapid review process to set and refine the review question, eligibility criteria, and the outcomes of interest [
      • Tricco A.C.
      • Zarin W.
      • Rios P.
      • Nincic V.
      • Khan P.A.
      • Ghassemi M.
      • et al.
      Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review.
      ,
      • Soobiah C.
      • Straus S.E.
      • Manley G.
      • Marr S.
      • Paus Jenssen E.
      • Teare S.
      • et al.
      Engaging knowledge users in a systematic review on the comparative effectiveness of geriatrician-led models of care is possible: a cross-sectional survey using the Patient Engagement Evaluation Tool.
      ]. The pandemic and its implications are rapidly evolving, making it difficult to clearly articulate and finalize the research question for examination. In some instances, decision-makers have asked researchers to proceed with COVID-related priority rapid reviews and given rapid review teams flexibility to identify and prioritize specific questions based on local/national/international needs [
      Cochrane Ireland.
      ,
      Cochrane France
      ]. In addition, decision-makers require rapid reviews to be completed faster than ever—often within five to 10 days [
      Cochrane's work on Rapid Reviews in response to COVID-19. Cochrane.
      ] and some within hours [].
      Other important stakeholders to partner with in the rapid review include clinicians and patients who can be involved with the entire rapid review process, and in particular, the selection of outcomes for inclusion in the review. It is important that rapid reviews address the need and include the views of front-line clinicians who are dealing with COVID-19. However, many clinicians are overwhelmed with clinical work due to COVID-19 and are unable to participate in scoping a rapid review as they might otherwise do. This might render the rapid review findings less relevant to clinical decision-making. Furthermore, in an era that has championed patients as partners in research, there is an imperative to involve patients and citizens as key stakeholders [
      Strategy for Patient-Oriented Research. Canadian Institutes of Health Research.
      ]. However, it is challenging to meaningfully engage patient partners within such limited timelines.
      Another problem is that there is little coordination of the evidence needs of decision-makers between local, provincial/regional, country and global levels, making it difficult for researchers, as well as research funders, to mitigate duplication [
      • Page M.J.
      • Welch V.A.
      • Haddaway N.R.
      • Karunananthan S.
      • Maxwell L.J.
      • Tugwell P.
      ‘One more time’: why replicating some syntheses of evidence relevant to COVID-19 makes sense.
      ] and research waste []. It is nearly impossible for a researcher to know whether the rapid review they are planning to conduct or were asked to conduct has already been completed. Indeed, there were 1,028 registered COVID-related protocols for systematic reviews of human studies in PROSPERO from March 1st to May 26th, 2020.

      2.1.2 Potential solutions

      Some organizations, such as the Cochrane Collaboration, are consulting with international partners (e.g., WHO) to prioritize rapid review topics [
      Cochrane's work on Rapid Reviews in response to COVID-19. Cochrane.
      ]. In addition to posting topics, it is essential to register rapid review protocols to decrease duplication and provide a publicly available record of the rapid review. Initiatives involving several knowledge synthesis organizations internationally are working together to reduce duplication of rapid reviews [
      COVID-END: COVID-19 evidence network to support decision-making. McMaster University.
      ,
      • Haddaway N.R.
      Open synthesis: on the need for evidence synthesis to embrace Open Science.
      ], as well as provide a list of resources that are updated continuously (e.g., COVID-END). Use of guidance and tools on rapid review conduct [
      ,
      Cochrane COVID rapid reviews.
      ] can help researchers methodologically tailor the methods for their rapid reviews thoughtfully.
      Experienced rapid review teams facilitate review conceptualization, as well as the entire process in a short period. For example, experienced groups will know how many team members are required to complete the rapid review within a specific timeframe. Furthermore, experienced teams can make thoughtful decisions about the scope of the project and whether it is important to include types of studies (e.g., randomized trials, systematic reviews, guidelines) or sources of evidence (e.g., websites, organizational policies and procedures, news sources, social media) to ensure the review is feasible and relevant. Such teams will also know when it is appropriate to limit the number of outcomes using COVID-19-specific core outcome sets [
      Strategy for Patient-Oriented Research. Canadian Institutes of Health Research.
      ].
      Teams can make use of collaborative tools, such as online meeting platforms and email, to interact with decision-makers to help conceptualize the rapid review. Some teams consult with experts to provide their insight on contextualizing the rapid review findings via quick telephone calls [
      Health Forum
      McMaster University.
      ] at the end of the review. The Cochrane COVID-19 consumer rapid response group is supporting teams and consumers to collaborate in the time available for completing a rapid review [
      Cochrane Consumer Network
      COVID-19 consumer rapid response group.
      ]. The process of involving consumers’ needs to be practical, articulated clearly, and in line with the rapid review timelines.

      2.1.3 Research needs

      Further research is required on how to meaningfully prioritize topics and outcomes included in rapid reviews, as well as engage a range of stakeholders in rapid reviews to ensure all perspectives are included.

      2.2 Conducting a literature search for COVID-19 rapid reviews

      2.2.1 Challenges

      A rapid review should include a comprehensive literature search [
      ] of multiple bibliographic databases (e.g., MEDLINE, Embase) [
      • Lasserson T.J.
      • Thomas J.
      • JPT H.
      Chapter 1: starting a review.
      ]. However, searching the gray literature (i.e., unpublished and difficult to locate material) has become increasingly important because emergent literature on COVID-19 is scattered across numerous sources, such as websites, public health guidelines, organizational policies and procedures, news sources, social media, clinical trials, COVID-19 repositories, and preprint servers (e.g., medRxiv). Searching these sources is complex because of a lack of indexing and poor functionality of the search interfaces. The data are geographically diverse and often written in languages other than English. As researchers are increasingly using alternative methods of sharing their research findings (e.g., ResearchGate, LinkedIn, preprint servers), librarians and information specialists must in turn adapt their typical search methods to account for this shift and may need to rapidly develop new skills.

      2.2.2 Potential solutions

      COVID-19 repositories and research/resource guides with lists of traditional and gray literature sources (e.g., WHO COVID-19, LitCovid, COVID-END) can be used to ensure the rapid review includes relevant studies that may not be captured by electronic databases [
      Guide to COVID-19 evidence sources. McMaster University.
      ]. For COVID-19 rapid reviews, studies in all languages should be considered for inclusion, which requires access to quick and trusted translations; membership in international networks such as Cochrane could help facilitate this. Teams can consider contacting their librarian as soon that they know a rapid review is on the horizon so that they can plan the literature search to meet the quick timeline. As well, based on our collective experience, some teams have prioritized specificity rather than sensitivity to make the literature searches more manageable for COVID-19 rapid reviews. Updating the literature search the same week as the rapid review becomes publicly available is one approach that ensures rapid reviews are up to date.

      2.2.3 Research needs

      The development and validation of a COVID-specific gray literature search will facilitate greater efficiency of literature searches for rapid reviews. A study on the trade-offs between taking the time to search for and appraise the quality of studies identified through alternative sources of evidence (i.e., preprints) and the impact on decision-making would provide useful information for rapid review conduct.

      2.3 Conducting screening, data abstraction, and assessment of methodological limitations for COVID-19 rapid reviews

      2.3.1 Challenges

      Teams may have to consider which way to streamline the screening and data abstraction methods for the rapid review conduct. It is unclear what the impact of using several different approaches such as semi-automated screening tools (e.g., EPPI Reviewer), crowdsourcing [
      • Nama N.
      • Barrowman N.
      • O'Hearn K.
      • Sampson M.
      • Zemek R.
      • McNally J.D.
      Quality control for crowdsourcing citation screening: the importance of assessment number and qualification set size.
      ,
      • Gartlehner G.
      • Affengruber L.
      • Titscher V.
      • Noel-Storr A.
      • Dooley G.
      • Ballarini N.
      • et al.
      Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial.
      ,
      • O’Hearn K.
      • Gertsman S.
      • Sampson M.
      • Webster R.J.
      • Tsampalieros A.
      • Ng R.
      • et al.
      Decontaminating N95 masks with Ultraviolet Germicidal Irradiation (UVGI) does not impair mask efficacy and safety: A Systematic Review.
      ,

      Gertsman S, Agarwal A, O'Hearn K, Webster RJ, Tsampalieros A, Barrowman N, et al. Microwave- and Heat-Based Decontamination of N95 Filtering Facepiece Respirators: A Systematic Review 2020. Available at: https://osf.io/4whsx/. Accessed May 25 2020.

      ,
      • Zorko D.J.
      • Gertsman S.
      • O'Hearn K.
      • Timmerman N.
      • Ambu-Ali N.
      • Dinh T.
      • et al.
      Decontamination Interventions for the Reuse of Surgical Mask Personal Protective Equipment: A Systematic Review.
      ,
      • O'Hearn K.
      • Gertsman S.
      • Webster R.J.
      • Tsampalieros A.
      • Ng R.
      • Gibson J.
      • et al.
      Efficacy and Safety of Disinfectants for Decontamination of N95 and SN95 Filtering Facepiece Respirators: A Systematic Review.
      ], or having only one person involved with the screening [
      • Clark J.
      • Glasziou P.
      • Del Mar C.
      • Bannach-Brown A.
      • Stehlik P.
      • Scott A.M.
      A full systematic review was completed in 2 weeks using automation tools: a case study.
      ] may have on results of the rapid review [
      • Robson R.C.
      • Pham B.
      • Hwee J.
      • Thomas S.M.
      • Rios P.
      • Page M.J.
      • et al.
      Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
      ,
      • Marshall I.J.
      • Marshall R.
      • Wallace B.C.
      • Brassey J.
      • Thomas J.
      Rapid reviews may produce different results to systematic reviews: a Meta-Epidemiological Study.
      ]. Similarly, it is unclear whether certain data extraction tools (with or without data mining features) are accurate and reliable. Related to appraisal of methodological limitations, the literature included in COVID-19 rapid reviews may be of lower methodological quality due to the rapid nature that the primary studies themselves have been conducted.

      2.3.2 Potential solutions

      No matter what approach, methods must be transparently reported and limitations need to be discussed. Some review groups are conducting data abstraction across multiple team members with live (synchronous) sharing of data or using crowdsourcing approaches. As well, the use of online software (e.g., DistillerSR, Covidence) and parallelization of tasks will make rapid reviews more efficient. Appraising the methodological limitations takes time yet can be incorporated into applying GRADE (or GRADE CERQual) of the evidence [
      Grade Working Group
      ]. This provides an indication of how trustworthy the rapid review results are and might be particularly important for COVID-19 rapid reviews, which often rely on non–peer-reviewed sources. However, more time might be required to appraise evidence from non–peer-reviewed sources. Some teams are limiting methodological assessments to only studies that are included in the analysis (whether qualitative or quantitative) to make the review more feasible.

      2.3.3 Research needs

      More research is required on whether the time tradeoff outweighs the potential of introducing error for various technological solutions for screening, data abstraction, and assessment of methodological limitations, such as crowdsourcing automation. This is particularly important for reviews that include many different types of study designs (e.g., qualitative and quantitative evidence) and sources of evidence (e.g., gray literature). The impact of including non–peer-reviewed sources (e.g., preprints) on the feasibility of the rapid review can be explored.

      2.4 Synthesis and interpretation of results for COVID-19 rapid reviews

      2.4.1 Challenges

      Because COVID-19 is an emerging disease, researchers may include indirect evidence from other pandemics or respiratory illnesses. However, it is challenging to extrapolate these findings to COVID-19, impacting interpretation of results. Furthermore, the outcomes examined in the primary studies included in the rapid review may vary, contributing heterogeneity and making any statistical pooling (e.g., meta-analysis) inappropriate. In addition, researchers are working at an incredibly fast pace and this makes it more challenging to interpret results in a thoughtful manner. The rapidity also makes it challenging to include the interpretation of results from all decision-makers.

      2.4.2 Potential solutions

      The interpretation of results needs to carefully consider any streamlined methods that were used. Researchers should be specific and transparent about what might have been lost in the process and what needs to be addressed in the future, perhaps through a more comprehensive review, and when such a review should be performed. If a meta-analysis was not feasible, it is important to report effect sizes with confidence intervals. In qualitative synthesis, it may not be possible to conduct subgroup analyses but this can be addressed in future updates. Some teams provide decision-makers with summary of findings tables without a descriptive writeup of results to facilitate completion in a shorter period. Working closely with decision-makers to interpret the rapid review results will ensure that the end product is relevant and fit for purpose (e.g., consulting experts to provide evidence contextualization at the review completion).

      2.4.3 Research needs

      Evaluation of technologies that automatically chart and summarize the content of included studies (e.g., technologies used to create a dashboard of research) [
      COVID-19 primer.
      ].

      2.5 Dissemination of COVID-19 rapid reviews

      2.5.1 Challenges

      The up-to-the-minute nature of the COVID-19 pandemic means that the traditional academic publishing model cannot keep up with the wave of evidence being produced. And decision-makers cannot wait for the rapid review to be published. At the same time, findings need to be presented in a complete and unbiased way and in a format that is clear to facilitate uptake of results. Developing a thoughtful plan for dissemination is even more challenging when the timeline is reduced leaving little time for communication or additional tasks.

      2.5.2 Potential solutions

      To quickly disseminate rapid review findings, researchers can consider other mechanisms, such as the Open Science Framework [
      OSFHome.
      ], Zenodo [
      Zenodo.
      ], or preprint servers. Use of short (e.g., 1-page) evidence summaries can [
      How to write a plain language summary of a Cochrane intervention review. Cochrane.
      ,
      How to write a plain language summary of a Cochrane EPOC Qualitative Evidence Synthesis. Cochrane.
      ] facilitate uptake of results with key messages highlighted upfront for the end user. Considering targeted dissemination mediums, such as infograms, podcasts, YouTube, LinkedIn, Twitter, ResearchGate, and media releases, might be required for dissemination above and beyond publishing in a peer-reviewed journal. Linkages with teams of data mobilizers and academic detailers, as well as communication teams, can facilitate dissemination of results. Use of evidence-informed dissemination strategies should be considered to ensure wide uptake of results [
      Launch of Cochrane’s dissemination Checklist and guidance. Cochrane.
      ].

      2.5.3 Research needs

      Evaluation of the impact of rapid review evidence in the decision-making process during COVID-19 through quantitative (e.g., surveys) and qualitative (e.g., interviews, focus groups) research including an array of decision-makers (e.g., citizens, public, health care providers, policy-makers).

      2.6 Updating COVID-19 rapid reviews

      2.6.1 Challenges

      Decision-makers are requesting rapid reviews that are updated on a continuous basis (i.e., living reviews [
      • Lasserson T.J.
      • Thomas J.
      • JPT H.
      Chapter 1: starting a review.
      ]), which is an emerging methodology. For example, it is unclear how often the literature should be searched and when the optimal timing of updates, or full reviews, should take place. Communication of updates regarding how the results and conclusion have changed is also challenging. As well, many teams do not have funding to conduct continuous updates of rapid reviews, creating a sustainability issue.

      2.6.2 Potential solutions

      Some teams are making use of automation in searching and screening to convert their rapid reviews into living rapid reviews. As well, some teams are working with their decision-makers to reconsider funding structures to allow living rapid reviews to be conducted on an ongoing basis during COVID-19. Organizations such as Cochrane and the Campbell Collaboration have processes in place for the regular updates of published reviews. This may be particularly helpful for authors of rapid reviews that may want to return at a later stage to continue their review.

      2.6.3 Research needs

      It is unclear whether some topics are more amenable to a living rapid review than others.

      3. Conclusion

      In summary, we have outlined several methodological challenges that have arisen during the conduct of COVID-19 rapid reviews. Although rapid reviews are being conducted quickly, we must use thoughtful methods and transparent reporting of our products [
      ,
      • Hernandez A.
      • Paeile C.
      • Perez H.
      • Pelissier T.
      • Soto-Moyano R.
      Kappa opioid receptor-mediated depression of activity evoked in convergent dorsal horn cells by thermal and non-thermal noxious stimulation.
      ]. It is hoped that the suggestions presented here are helpful for the conduct of COVID-19 rapid reviews and can be used in future times of disaster and emergency.

      CRediT authorship contribution statement

      Andrea C. Tricco: Conceptualization, Methodology, Writing - original draft, Writing - review & editing. Chantelle M. Garritty: Writing - original draft, Writing - review & editing. Leah Boulos: Writing - original draft, Writing - review & editing. Craig Lockwood: Writing - original draft, Writing - review & editing. Michael Wilson: Writing - original draft, Writing - review & editing. Jessie McGowan: Writing - original draft, Writing - review & editing. Michael McCaul: Writing - original draft, Writing - review & editing. Brian Hutton: Writing - original draft, Writing - review & editing. Fiona Clement: Writing - review & editing. Nicole Mittmann: Writing - review & editing. Declan Devane: Writing - review & editing. Etienne V. Langlois: Writing - review & editing. Ahmed M. Abou-Setta: Writing - review & editing. Catherine Houghton: Writing - review & editing. Claire Glenton: Writing - review & editing. Shannon E. Kelly: Writing - review & editing. Vivian A. Welch: Writing - review & editing. Annie LeBlanc: Writing - review & editing. George A. Wells: Writing - review & editing. Ba’ Pham: Writing - review & editing. Simon Lewin: Writing - review & editing. Sharon E. Straus: Writing - review & editing.

      Acknowledgments

      The authors thank Shazia Siddiqui for her help with formatting the paper and compiling the ideas from the coauthors.

      Supplementary data

      References

        • Hartling L.
        • Guise J.M.
        • Kato E.
        • Anderson J.
        • Aronson N.
        • Belinson S.
        • et al.
        EPC Methods: An Exploration of Methods and Context for the Production of Rapid Reviews.
        Agency for Healthcare Research and Quality (US), Rockville, MD2015
      1. Cochrane's work on Rapid Reviews in response to COVID-19. Cochrane.
        (Available at)
      2. Tricco A.C. Langlois E.V. Straus S.E. Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization, Geneva2017 (Licence: CC BY-NC-SA 3.0 IGO)
        • Langlois E.V.
        • Straus S.E.
        • Antony J.
        • King V.J.
        • Tricco A.C.
        Using rapid reviews to strengthen health policy and systems and progress towards universal health coverage.
        BMJ Glob Health. 2019; 4: e001178
      3. COVID-END: COVID-19 evidence network to support decision-making. McMaster University.
        (Available at)
        • Tricco A.C.
        • Zarin W.
        • Rios P.
        • Nincic V.
        • Khan P.A.
        • Ghassemi M.
        • et al.
        Engaging policy-makers, health system managers, and policy analysts in the knowledge synthesis process: a scoping review.
        Implement Sci. 2018; 13: 31
        • Soobiah C.
        • Straus S.E.
        • Manley G.
        • Marr S.
        • Paus Jenssen E.
        • Teare S.
        • et al.
        Engaging knowledge users in a systematic review on the comparative effectiveness of geriatrician-led models of care is possible: a cross-sectional survey using the Patient Engagement Evaluation Tool.
        J Clin Epidemiol. 2019; 113: 58-63
      4. Cochrane Ireland.
        (Available at)
        https://ireland.cochrane.org
        Date accessed: May 26, 2020
        • Cochrane France
        (Available at)
        https://france.cochrane.org
        Date accessed: May 26, 2020
        • McMaster Health Forum
        McMaster University.
        (Available at)
      5. Strategy for Patient-Oriented Research. Canadian Institutes of Health Research.
        (Available at)
        https://cihr-irsc.gc.ca/e/41204.html
        Date accessed: May 26, 2020
        • Page M.J.
        • Welch V.A.
        • Haddaway N.R.
        • Karunananthan S.
        • Maxwell L.J.
        • Tugwell P.
        ‘One more time’: why replicating some syntheses of evidence relevant to COVID-19 makes sense.
        J Clin Epidemiol. 2020;
      6. Lancet. 2019; 393 (Available at): 493-610.e6-e28
        • Haddaway N.R.
        Open synthesis: on the need for evidence synthesis to embrace Open Science.
        Environ Evid. 2018; 7: 26
      7. Cochrane COVID rapid reviews.
        (Available at)
        • Health Forum
        McMaster University.
        (Available at)
        https://www.mcmasterforum.org
        Date accessed: May 26, 2020
        • Cochrane Consumer Network
        COVID-19 consumer rapid response group.
        (Available at)
        • Lasserson T.J.
        • Thomas J.
        • JPT H.
        Chapter 1: starting a review.
        in: Higgins J.P.T. Thomas J. Chandler J. Cumpston M. Li T. Page M.J. Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). Cochrane. 2019 (Available at)
        www.training.cochrane.org/handbook
        Date accessed: July 5, 2020
      8. Guide to COVID-19 evidence sources. McMaster University.
        (Available at)
        https://www.mcmasterforum.org/find-evidence/guide-to-covid-19-evidence-sources
        Date accessed: May 5, 2020
        • Nama N.
        • Barrowman N.
        • O'Hearn K.
        • Sampson M.
        • Zemek R.
        • McNally J.D.
        Quality control for crowdsourcing citation screening: the importance of assessment number and qualification set size.
        J Clin Epidemiol. 2020; 122: 160-162
        • Gartlehner G.
        • Affengruber L.
        • Titscher V.
        • Noel-Storr A.
        • Dooley G.
        • Ballarini N.
        • et al.
        Single-reviewer abstract screening missed 13 percent of relevant studies: a crowd-based, randomized controlled trial.
        J Clin Epidemiol. 2020; 121: 20-28
        • O’Hearn K.
        • Gertsman S.
        • Sampson M.
        • Webster R.J.
        • Tsampalieros A.
        • Ng R.
        • et al.
        Decontaminating N95 masks with Ultraviolet Germicidal Irradiation (UVGI) does not impair mask efficacy and safety: A Systematic Review.
        Open Science Framework (OSF), 2020
        https://osf.io/29z6u/
        Date accessed: April 11, 2020
      9. Gertsman S, Agarwal A, O'Hearn K, Webster RJ, Tsampalieros A, Barrowman N, et al. Microwave- and Heat-Based Decontamination of N95 Filtering Facepiece Respirators: A Systematic Review 2020. Available at: https://osf.io/4whsx/. Accessed May 25 2020.

        • Zorko D.J.
        • Gertsman S.
        • O'Hearn K.
        • Timmerman N.
        • Ambu-Ali N.
        • Dinh T.
        • et al.
        Decontamination Interventions for the Reuse of Surgical Mask Personal Protective Equipment: A Systematic Review.
        2020 (Available at:)
        https://osf.io/z7exu/
        Date accessed: May 25, 2020
        • O'Hearn K.
        • Gertsman S.
        • Webster R.J.
        • Tsampalieros A.
        • Ng R.
        • Gibson J.
        • et al.
        Efficacy and Safety of Disinfectants for Decontamination of N95 and SN95 Filtering Facepiece Respirators: A Systematic Review.
        2020 (Available at:)
        https://osf.io/ct6m8/
        Date accessed: May 25, 2020
        • Clark J.
        • Glasziou P.
        • Del Mar C.
        • Bannach-Brown A.
        • Stehlik P.
        • Scott A.M.
        A full systematic review was completed in 2 weeks using automation tools: a case study.
        J Clin Epidemiol. 2020; 121: 81-90
        • Robson R.C.
        • Pham B.
        • Hwee J.
        • Thomas S.M.
        • Rios P.
        • Page M.J.
        • et al.
        Few studies exist examining methods for selecting studies, abstracting data, and appraising quality in a systematic review.
        J Clin Epidemiol. 2019; 106: 121-135
        • Marshall I.J.
        • Marshall R.
        • Wallace B.C.
        • Brassey J.
        • Thomas J.
        Rapid reviews may produce different results to systematic reviews: a Meta-Epidemiological Study.
        J Clin Epidemiol. 2019; 109: 30-41
        • Grade Working Group
        (Available at)
        https://www.gradeworkinggroup.org
        Date accessed: May 26, 2020
      10. COVID-19 primer.
        (Available at)
        https://covid19primer.com/dashboard
        Date accessed: May 25, 2020
      11. OSFHome.
        (Available at)
        https://osf.io
        Date accessed: May 25, 2020
      12. Zenodo.
        (Available at)
        https://zenodo.org
        Date accessed: May 20, 2020
      13. How to write a plain language summary of a Cochrane intervention review. Cochrane.
        (Available at)
      14. How to write a plain language summary of a Cochrane EPOC Qualitative Evidence Synthesis. Cochrane.
        (Available at)
      15. Launch of Cochrane’s dissemination Checklist and guidance. Cochrane.
        (Available at)
        • Hernandez A.
        • Paeile C.
        • Perez H.
        • Pelissier T.
        • Soto-Moyano R.
        Kappa opioid receptor-mediated depression of activity evoked in convergent dorsal horn cells by thermal and non-thermal noxious stimulation.
        Eur J Pharmacol. 1990; 186: 323-325