Same family, different species: methodological conduct and quality varies according to purpose for five types of knowledge synthesis

  • Andrea C. Tricco
    Correspondence
    Corresponding author. Tel.: +1-416-864-6060; fax: +1-416-864-5805.
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada

    Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, 6th Floor, Toronto, Ontario M5T 3M7, Canada
    Search for articles by this author
  • Wasifa Zarin
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Marco Ghassemi
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Vera Nincic
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Erin Lillie
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Matthew J. Page
    Affiliations
    School of Public Health & Preventive Medicine, Monash University, 553 St Kilda Road, Melbourne, Victoria 3004, Australia

    School of Social and Community Medicine, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol BS8 2PS, UK
    Search for articles by this author
  • Larissa Shamseer
    Affiliations
    Ottawa Hospital Research Institute, The Ottawa Hospital, 501 Smyth Road, PO BOX 201B, Ottawa, Ontario K1H 8L6, Canada
    Search for articles by this author
  • Jesmin Antony
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Patricia Rios
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • Jeremiah Hwee
    Affiliations
    Epidemiology Division, Dalla Lana School of Public Health, University of Toronto, 155 College Street, 6th Floor, Toronto, Ontario M5T 3M7, Canada
    Search for articles by this author
  • Areti Angeliki Veroniki
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada
    Search for articles by this author
  • David Moher
    Affiliations
    Ottawa Hospital Research Institute, The Ottawa Hospital, 501 Smyth Road, PO BOX 201B, Ottawa, Ontario K1H 8L6, Canada
    Search for articles by this author
  • Lisa Hartling
    Affiliations
    School of Public Health, University of Alberta, 116 St & 85 Ave, Edmonton, Alberta T6G 2R3, Canada
    Search for articles by this author
  • Ba' Pham
    Affiliations
    Toronto Health Economics and Technology Assessment, University of Toronto, 27 Kings College Circle, Toronto, Ontario M5S 1A1, Canada
    Search for articles by this author
  • Sharon E. Straus
    Affiliations
    Knowledge Translation Program, Li Ka Shing Knowledge Institute, St. Michael's Hospital, 209 Victoria Street, East Building, Toronto, Ontario M5B 1T8, Canada

    Department of Geriatric Medicine, University of Toronto, 27 Kings College Circle, Toronto, Ontario M5S 1A1, Canada
    Search for articles by this author
Published:November 02, 2017DOI:https://doi.org/10.1016/j.jclinepi.2017.10.014

      Abstract

      Objectives

      The aim of the study was to characterize methodological conduct, reporting, and quality of five knowledge synthesis (KS) approaches.

      Study Design and Setting

      Retrospective analysis of a convenience sample of five published databases of KS approaches: overview of reviews (n = 74), scoping reviews (n = 494), rapid reviews (n = 84), systematic reviews (n = 300), and network meta-analyses (NMAs; n = 456). Data in the five published databases were abstracted by two reviewers independently, any missing data for this retrospective analysis were abstracted by one experienced reviewer. Methods were appraised using the A MeaSurement Tool to Assess systematic Reviews (AMSTAR) tool. Descriptive analysis was performed.

      Results

      Reporting the use of a protocol ranged from 4% for rapid reviews to 32% for systematic reviews. The use of two reviewers for citation and full-text screening ranged from 20% for scoping reviews to 60% for NMAs. Data abstraction was performed in duplicate for 11% of rapid reviews and 54% of NMAs, and for risk of bias appraisal, this ranged from 6% for scoping reviews to 41% for NMAs. NMAs had the highest median percentage of maximum obtainable AMSTAR score (64%; Q1–Q3:45–73%), while scoping reviews had the lowest (25%; Q1–Q3:13–38%).

      Conclusion

      NMAs consistently scored the highest on the AMSTAR tool likely because the purpose is to estimate treatment effects statistically. Scoping reviews scored the lowest (even after adjusting the score for not relevant items) likely because the purpose is to characterize the literature.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Grimshaw J.A.
        A Guide to Knowledge Synthesis: A Knowledge Synthesis Chapter.
        Canadian Institute of Health Research, Ottawa, Canada, 2010
      1. Higgins J. Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, 2011 ([updated March 2011]. Available at http://handbook.cochrane.org. Accessed August 23, 2017)
        • Straus S.E.
        • Kastner M.
        • Soobiah C.
        • Antony J.
        • Tricco A.C.
        Introduction: Engaging researchers on developing, using, and improving knowledge synthesis methods: a series of articles describing the results of a scoping review on emerging knowledge synthesis methods.
        J Clin Epidemiol. 2016; 73: 15-18
        • Tricco A.C.
        • Soobiah C.
        • Antony J.
        • Cogo E.
        • MacDonald H.
        • Lillie E.
        • et al.
        A scoping review identifies multiple emerging knowledge synthesis methods, but few studies operationalize the method.
        J Clin Epidemiol. 2016; 73: 19-28
        • Tricco A.C.
        • Antony J.
        • Soobiah C.
        • Kastner M.
        • MacDonald H.
        • Cogo E.
        • et al.
        Knowledge synthesis methods for integrating qualitative and quantitative data: a scoping review reveals poor operationalization of the methodological steps.
        J Clin Epidemiol. 2016; 73: 29-35
        • Tricco A.C.
        • Antony J.
        • Soobiah C.
        • Kastner M.
        • Cogo E.
        • MacDonald H.
        • et al.
        Knowledge synthesis methods for generating or refining theory: a scoping review reveals that little guidance is available.
        J Clin Epidemiol. 2016; 73: 36-42
        • Kastner M.
        • Antony J.
        • Soobiah C.
        • Straus S.E.
        • Tricco A.C.
        Conceptual recommendations for selecting the most appropriate knowledge synthesis method to answer research questions related to complex evidence.
        J Clin Epidemiol. 2016; 73: 43-49
        • Perrier L.
        • Lightfoot D.
        • Kealey M.R.
        • Straus S.E.
        • Tricco A.C.
        Knowledge synthesis research: a bibliometric analysis.
        J Clin Epidemiol. 2016; 73: 50-57
        • Arksey H.
        • O'Malley L.
        Scoping studies: towards a methodological framework.
        Int J Soc Res Methodol. 2005; 8: 19-32
        • Hartling L.
        • Chisholm A.
        • Thomson D.
        • Dryden D.M.
        A descriptive analysis of overviews of reviews published between 2000 and 2011.
        PLoS One. 2012; 7: e49667
        • Ballard M.
        • Montgomery P.
        Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist.
        Res Synth Methods. 2017; 8: 92-108
        • Moja L.
        • Fernandez del Rio M.P.
        • Banzi R.
        • Cusi C.
        • D'Amico R.
        • Liberati A.
        • et al.
        Multiple systematic reviews: methods for assessing discordances of results.
        Intern Emerg Med. 2012; 7: 563-568
        • Khangura S.
        • Konnyu K.
        • Cushman R.
        • Grimshaw J.
        • Moher D.
        Evidence summaries: the evolution of a rapid review approach.
        Syst Rev. 2012; 1: 1
        • Ismail S.A.
        • Abbara A.
        • Collin S.M.
        • Orcutt M.
        • Coutts A.P.
        • Maziak W.
        • et al.
        Communicable disease surveillance and control in the context of conflict and mass displacement in Syria.
        Int J Infect Dis. 2016; 47: 15-22
        • Bucher H.C.
        • Guyatt G.H.
        • Griffith L.E.
        • Walter S.D.
        The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials.
        J Clin Epidemiol. 1997; 50: 683-691
        • Moher D.
        • Stewart L.
        • Shekelle P.
        All in the Family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more.
        Syst Rev. 2015; 4: 183
        • World Health Organization
        Frequently Asked questions.
        2017 (Available at http://www.who.int/suggestions/faq/en/. Accessed August 23, 2017)
        • Tricco A.C.
        • Lillie E.
        • Zarin W.
        • O'Brien K.
        • Colquhoun H.
        • Kastner M.
        • et al.
        A scoping review on the conduct and reporting of scoping reviews.
        BMC Med Res Methodol. 2016; 16: 15
        • Tricco A.C.
        • Antony J.
        • Zarin W.
        • Strifler L.
        • Ghassemi M.
        • Ivory J.
        • et al.
        A scoping review of rapid review methods.
        BMC Med. 2015; 13: 224
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of Biomedical research: a Cross-Sectional study.
        PLoS Med. 2016; 13: e1002028
        • Petropoulou M.
        • Nikolakopoulou A.
        • Veroniki A.A.
        • Rios P.
        • Vafaei A.
        • Zarin W.
        • et al.
        Bibliographic study showed improving statistical methodology of network meta-analyses published between 1999 and 2015.
        J Clin Epidemiol. 2017; 82: 20-28
        • Zarin W.
        • Veroniki A.A.
        • Nincic V.
        • Vafaei A.
        • Reynen E.
        • Motiwala S.S.
        • et al.
        Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review.
        BMC Med. 2017; 15: 3
        • Shamseer L.
        • Moher D.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation.
        BMJ. 2015; 349: g7647
        • Pollock M.
        • Fernandes R.M.
        • Becker L.A.
        • Featherstone R.
        • Hartling L.
        What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary.
        Syst Rev. 2016; 5: 190
      2. International Classification of Diseases and Related Health Problems 10th Revision. World Health Organization, Geneva, Switzerland, 2016
        • Shea B.J.
        • Hamel C.
        • Wells G.A.
        • Bouter L.M.
        • Kristjansson E.
        • Grimshaw J.
        • et al.
        AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
        J Clin Epidemiol. 2009; 62: 1013-1020
        • Schuit E.
        • Ioannidis J.P.
        Network meta-analyses performed by contracting companies and commissioned by industry.
        Syst Rev. 2016; 5: 198
        • R Core Team
        R: A language and environment for statistical computing.
        R Foundation for Statistical Computing, Vienna, Austria2016
        • Wickham H.
        • Chang W.
        R Studio. Package ‘ggplot2’. An implementation of the grammar of graphics. 2.1.0.
        2016 (Available at https://cran.r-project.org/web/packages/ggplot2/ggplot2.pdf. Accessed August 23, 2017)
        • Knowledge Translation Program
        The KT program.
        2016 (Available at http://knowledgetranslation.net/. Accessed August 23, 2017)
        • Emilsson L.
        • Holme O.
        • Bretthauer M.
        • Cook N.R.
        • Buring J.E.
        • Loberg M.
        • et al.
        Systematic review with meta-analysis: the comparative effectiveness of aspirin vs. screening for colorectal cancer prevention.
        Aliment Pharmacol Ther. 2017; 45: 193-204
        • Saborido C.M.
        • Hockenhull J.
        • Bagust A.
        • Boland A.
        • Dickson R.
        • Todd D.
        Systematic review and cost-effectiveness evaluation of 'pill-in-the-pocket' strategy for paroxysmal atrial fibrillation compared to episodic in-hospital treatment or continuous antiarrhythmic drug therapy.
        Health Technol Assess. 2010; 14: 1-75
        • Best A.
        • Greenhalgh T.
        • Lewis S.
        • Saul J.E.
        • Carroll S.
        • Bitz J.
        Large-system transformation in health care: a realist review.
        Milbank Q. 2012; 90: 421-456
        • Jepson R.
        • Di Blasi Z.
        • Wright K.
        • Ter Riet G.
        Scoping review of the effectiveness of mental health services.
        University of York, York, UK, 2001
        • Futamura M.
        • Thomas K.S.
        • Grindlay D.J.
        • Doney E.J.
        • Torley D.
        • Williams H.C.
        Mapping systematic reviews on atopic eczema–an essential resource for dermatology professionals and researchers.
        PLoS One. 2013; 8: e58484
        • Jaana M.
        • Vartak S.
        • Ward M.M.
        Evidence-based health care management: what is the research evidence available for health care managers?.
        Eval Health Prof. 2014; 37: 314-334
        • Knight R.
        • Small W.
        • Pakula B.
        • Thomson K.
        • Shoveller J.
        A scoping study to identify opportunities to advance the ethical implementation and scale-up of HIV treatment as prevention: priorities for empirical research.
        BMC Med Ethics. 2014; 15: 54
        • McKinstry C.
        • Brown T.
        • Gustafsson L.
        Scoping reviews in occupational therapy: the what, why, and how to.
        Aust Occup Ther J. 2014; 61: 58-66
        • Mickan S.
        • Tilson J.K.
        • Atherton H.
        • Roberts N.W.
        • Heneghan C.
        Evidence of effectiveness of health care professionals using handheld computers: a scoping review of systematic reviews.
        J Med Internet Res. 2013; 15: e212
        • Samaan Z.
        • Mbuagbaw L.
        • Kosa D.
        • Borg Debono V.
        • Dillenburg R.
        • Zhang S.
        • et al.
        A systematic scoping review of adherence to reporting guidelines in health care literature.
        J Multidiscip Healthc. 2013; 6: 169-188
        • Sarrami-Foroushani P.
        • Travaglia J.
        • Debono D.
        • Braithwaite J.
        Key concepts in consumer and community engagement: a scoping meta-review.
        BMC Health Serv Res. 2014; 14: 250
        • van de Glind E.M.
        • van Enst W.A.
        • van Munster B.C.
        • Olde Rikkert M.G.
        • Scheltens P.
        • Scholten R.J.
        • et al.
        Pharmacological treatment of dementia: a scoping review of systematic reviews.
        Dement Geriatr Cogn Disord. 2013; 36: 211-228
        • Foster C.
        • Amir Z.
        • Jarrett N.
        • Stamataki Z.
        • Brearley S.
        • Scott I.
        Determining research priorities for cancer survivorship: consultation and evidence review.
        National Cancer Survivorship Initiative, UK, 2009
        • Parker G.
        • Arksey H.
        • Harden M.
        Meta-review of international evidence on interventions to support carers.
        Social Policy Research Unit, University of York, Heslington, England, 2010
        • MacDougall A.C.
        • Cobban S.J.
        • Compton S.M.
        Is periodontal disease related to adverse pregnancy outcomes? A scoping review.
        Can J Dent Hyg. 2011; 45: 53-60
        • Boyd A.
        • Chambers N.
        • French S.
        • King R.
        • Shaw D.
        • Whitehead A.
        A scoping study of emergency planning and management in health care: What further research is needed. Final report.
        NIHR Health Services and Delivery Research programme, 2012
        • Macpherson K.
        Screening hips of newborns in Scotland.
        National Health Services (NHS) Quality Improvement, Scotland, 2006
        • Tricco A.
        • Rios P.
        • Ashoor H.
        • Thomas S.
        • Zarin W.
        • Cardoso R.
        • et al.
        Systematic prospective assessment of rapid knowledge synthesis – SPARKS study.
        2017 (Available at https://osf.io/986pc/. Accessed August 23, 2017)
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        Ann Intern Med. 2009; 151: 264-269