GRADE guidelines: 21 part 1. Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy

  • Holger J. Schünemann
    Correspondence
    Corresponding author. Department of Health Research Methods, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada. Tel.: +1 905-525-9140x24931; fax: +1-905-522-9507.
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada
    Search for articles by this author
  • Reem A. Mustafa
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, University of Kansas Medical Center, Kansas City, KS, USA
    Search for articles by this author
  • Jan Brozek
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada
    Search for articles by this author
  • Karen R. Steingart
    Affiliations
    Department of Clinical Sciences, Liverpool School of Tropical Medicine, Pembroke Place, Liverpool L3 5QA, UK
    Search for articles by this author
  • Mariska Leeflang
    Affiliations
    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centers, Room J1b-214, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
    Search for articles by this author
  • Mohammad Hassan Murad
    Affiliations
    Division of Preventive Medicine, Mayo Clinic, 200 1st Street SW, Rochester, MN 55902, USA
    Search for articles by this author
  • Patrick Bossuyt
    Affiliations
    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centers, Room J1b-214, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
    Search for articles by this author
  • Paul Glasziou
    Affiliations
    CREBP, Faculty Health Science & Medicine, Bond University, Gold Coast QLD 4229, Australia
    Search for articles by this author
  • Roman Jaeschke
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada
    Search for articles by this author
  • Stefan Lange
    Affiliations
    Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen, Institute for Quality and Efficiency in Health Care (IQWiG), Im Mediapark 8, 50670 Köln, Germany Cologne, Germany
    Search for articles by this author
  • Joerg Meerpohl
    Affiliations
    Institute for Evidence in Medicine, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Freiburg, Germany

    Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany
    Search for articles by this author
  • Miranda Langendam
    Affiliations
    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam University Medical Centers, Room J1b-214, Meibergdreef 9, 1105 AZ Amsterdam, The Netherlands
    Search for articles by this author
  • Monica Hultcrantz
    Affiliations
    Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden
    Search for articles by this author
  • Gunn E. Vist
    Affiliations
    Norwegian Knowledge Centre for the Health Services, PO Box 7004, St Olavs Plass, 0130 Oslo, Norway
    Search for articles by this author
  • Elie A. Akl
    Affiliations
    Department of Internal Medicine, American University of Beirut, Riad-El-Solh Beirut, Beirut 1107 2020, Lebanon
    Search for articles by this author
  • Mark Helfand
    Affiliations
    Oregon Evidence-based Practice Center, Oregon Health & Science University, Portland VA Medical Center, Portland, OR, USA
    Search for articles by this author
  • Nancy Santesso
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada
    Search for articles by this author
  • Lotty Hooft
    Affiliations
    Cochrane Netherlands/Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands
    Search for articles by this author
  • Rob Scholten
    Affiliations
    Cochrane Netherlands/Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, P.O. Box 85500, 3508 GA Utrecht, The Netherlands
    Search for articles by this author
  • Måns Rosen
    Affiliations
    Swedish Agency for Health Technology Assessment and Assessment of Social Services (SBU), S:t Eriksgatan 117, SE-102 33, Stockholm, Sweden
    Search for articles by this author
  • Anne Rutjes
    Affiliations
    Clinical Trial Unit (CTU) Bern, Institute of Primary Health Care, University of Bern, Bern, Switzerland
    Search for articles by this author
  • Mark Crowther
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada
    Search for articles by this author
  • Paola Muti
    Affiliations
    Department of Oncology, McMaster University, 711 Concession Street, Hamilton, Ontario L8V1C3, Canada
    Search for articles by this author
  • Heike Raatz
    Affiliations
    University of Basel, Klingelbergstrasse 61, CH-4056 Basel, Switzerland

    Kleijnen Systematic Reviews Ltd, 6 Escrick Business Park, Escrick, York YO19 6FD, UK
    Search for articles by this author
  • Mohammed T. Ansari
    Affiliations
    School of Epidemiology and Public Health, Faculty of Medicine, Ottawa, Canada
    Search for articles by this author
  • John Williams
    Affiliations
    Duke University Medical Center and Durham Veterans Affairs Center for Health Services Research in Primary Care Durham, NC 27705, USA
    Search for articles by this author
  • Regina Kunz
    Affiliations
    Basel Institute of Clinical Epidemiology, University Hospital Basel, Hebelstrasse 10, Basel 4031, Switzerland
    Search for articles by this author
  • Jeff Harris
    Affiliations
    Harris Associates, 386 Richardson Way, Mill Valley, CA 94941, USA
    Search for articles by this author
  • Ingrid Arévalo Rodriguez
    Affiliations
    Clinical Biostatistics Unit, Hospital Universitario Ramón y Cajal, IRYCIS, CIBER of Epidemiology and Public Health, Madrid, Spain

    Centro de investigación en Salud Pública y Epidemiología Clínica, Facultad de Ciencias de la Salud Eugenio Espejo, Universidad UTE, Quito, Ecuador
    Search for articles by this author
  • Mikashmi Kohli
    Affiliations
    Department of Epidemiology, Biostatistics and Occupational Health, McGill University, 1650 Cedar Avenue, Montreal, Quebec H3G 1A4, Canada
    Search for articles by this author
  • Gordon H. Guyatt
    Affiliations
    Department of Health Research Methods, Evidence, and Impact, McMaster GRADE Centre, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S4K1, Canada

    Department of Medicine, University of Kansas Medical Center, Kansas City, KS, USA
    Search for articles by this author
  • for theGRADE Working Group
Published:February 11, 2020DOI:https://doi.org/10.1016/j.jclinepi.2019.12.020

      Abstract

      Objectives

      This article provides updated GRADE guidance about how authors of systematic reviews and health technology assessments and guideline developers can assess the results and the certainty of evidence (also known as quality of the evidence or confidence in the estimates) of a body of evidence addressing test accuracy (TA).

      Study Design and Setting

      We present an overview of the GRADE approach and guidance for rating certainty in TA in clinical and public health and review the presentation of results of a body of evidence regarding tests. Part 1 of the two parts in this 21st guidance article about how to apply GRADE focuses on understanding study design issues in test accuracy, provide an overview of the domains, and describe risk of bias and indirectness specifically.

      Results

      Supplemented by practical examples, we describe how raters of the evidence using GRADE can evaluate study designs focusing on tests and how they apply the GRADE domains risk of bias and indirectness to a body of evidence of TA studies.

      Conclusion

      Rating the certainty of a body of evidence using GRADE in Cochrane and other reviews and World Health Organization and other guidelines dealing with in TA studies helped refining our approach. The resulting guidance will help applying GRADE successfully for questions and recommendations focusing on tests.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Guyatt G.
        • Oxman A.D.
        • Akl E.A.
        • Kunz R.
        • Vist G.
        • Brozek J.
        • et al.
        GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Guyatt G.H.
        • Oxman A.D.
        • Schunemann H.J.
        • Tugwell P.
        • Knotterus A.
        GRADE guidelines: a new series of articles in the Journal of Clinical Epidemiology.
        J Clin Epidemiol. 2010; 64: 380-382
        • Puhan M.A.
        • Schunemann H.J.
        • Murad M.H.
        • Li T.
        • Brignardello-Petersen R.
        • Singh J.A.
        • et al.
        A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis.
        BMJ. 2014; 349: g5630
        • Schunemann H.J.
        • Best D.
        • Vist G.
        • Oxman A.D.
        • Group G.W.
        Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
        CMAJ. 2003; 169: 677-680
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Bossuyt P.
        • Chang S.
        • et al.
        GRADE: assessing the quality of evidence for diagnostic recommendations.
        ACP J Club. 2008; 149: 2
        • Spencer F.A.
        • Iorio A.
        • You J.
        • Murad M.H.
        • Schunemann H.J.
        • Vandvik P.O.
        • et al.
        Uncertainties in baseline risk estimates and confidence in treatment effects.
        BMJ. 2012; 345: e7401
        • Schunemann H.J.
        • Oxman A.D.
        • Brozek J.
        • Glasziou P.
        • Jaeschke R.
        • Vist G.E.
        • et al.
        Grading quality of evidence and strength of recommendations for diagnostic tests and strategies.
        BMJ. 2008; 336: 1106-1110
        • Guyatt G.
        • Oxman A.D.
        • Sultan S.
        • Brozek J.
        • Glasziou P.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes.
        J Clin Epidemiol. 2013; 66: 151-157
        • Alonso-Coello P.
        • Oxman A.D.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 2: clinical practice guidelines.
        BMJ. 2016; 353: i2089
        • Alonso-Coello P.
        • Schunemann H.J.
        • Moberg J.
        • Brignardello-Petersen R.
        • Akl E.A.
        • Davoli M.
        • et al.
        GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction.
        BMJ. 2016; 353: i2016
        • Schunemann H.J.
        • Mustafa R.
        • Brozek J.
        • Santesso N.
        • Alonso-Coello P.
        • Guyatt G.
        • et al.
        GRADE Guidelines: 16. GRADE evidence to decision frameworks for tests in clinical practice and public health.
        J Clin Epidemiol. 2016; 76: 89-98
        • Burford B.J.
        • Rehfuess E.
        • Schunemann H.J.
        • Akl E.A.
        • Waters E.
        • Armstrong R.
        • et al.
        Assessing evidence in public health: the added value of GRADE.
        J Public Health (Oxf). 2012; 34: 631-635
        • Thayer K.A.
        • Schunemann H.J.
        Using GRADE to respond to health questions with different levels of urgency.
        Environ Int. 2016; 92-93: 585-589
        • Schunemann H.J.
        • Hill S.R.
        • Kakad M.
        • Vist G.E.
        • Bellamy R.
        • Stockman L.
        • et al.
        Transparent development of the WHO rapid advice guidelines.
        PLoS Med. 2007; 4: e119
        • Iorio A.
        • Spencer F.A.
        • Falavigna M.
        • Alba C.
        • Lang E.
        • Burnand B.
        • et al.
        Use of GRADE for assessment of evidence about prognosis: rating confidence in estimates of event rates in broad categories of patients.
        BMJ. 2015; 350: h870
        • Guyatt G.H.
        • Ebrahim S.
        • Alonso-Coello P.
        • Johnston B.C.
        • Mathioudakis A.G.
        • Briel M.
        • et al.
        GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence.
        J Clin Epidemiol. 2017; 87: 14-22
        • Morgan R.L.
        • Thayer K.A.
        • Bero L.
        • Bruce N.
        • Falck-Ytter Y.
        • Ghersi D.
        • et al.
        GRADE: assessing the quality of evidence in environmental and occupational health.
        Environ Int. 2016; 92-93: 611-616
        • Zhang Y.
        • Alonso Coello P.
        • Guyatt G.
        • Yepes-Nunez J.J.
        • Akl E.A.
        • Hazlewood G.
        • et al.
        GRADE Guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences - inconsistency, imprecision, and other domains.
        J Clin Epidemiol. 2018; 111: 83-93
        • Zhang Y.
        • Alonso-Coello P.
        • Guyatt G.H.
        • Yepes-Nunez J.J.
        • Akl E.A.
        • Hazlewood G.
        • et al.
        GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-risk of bias and indirectness.
        J Clin Epidemiol. 2018; 111: 94-104
        • Brozek J.L.
        • Akl E.A.
        • Jaeschke R.
        • Lang D.M.
        • Bossuyt P.
        • Glasziou P.
        • et al.
        Grading quality of evidence and strength of recommendations in clinical practice guidelines: Part 2 of 3. The GRADE approach to grading quality of evidence about diagnostic tests and strategies.
        Allergy. 2009; 64: 1109-1116
        • Schunemann H.J.
        • Mustafa R.A.
        • Brozek J.
        • Steingart K.
        • Leeflang M.
        • Murad H.M.
        • et al.
        GRADE guidelines: 21 part 2. Inconsistency, Imprecision, publication bias and other domains for rating the certainty of evidence for test accuracy and presenting it in evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2020; ([Epub ahead of print])https://doi.org/10.1016/j.jclinepi.2019.12.021
        • Deeks J.J.
        Systematic reviews in health care: systematic reviews of evaluations of diagnostic and screening tests.
        BMJ. 2001; 323: 157-162
        • Schunemann H.J.
        • Mustafa R.A.
        • Brozek J.
        • Santesso N.
        • Bossuyt P.M.
        • Steingart K.R.
        • et al.
        GRADE guidelines: 22. The GRADE approach for tests and strategies-from test accuracy to patient-important outcomes and recommendations.
        J Clin Epidemiol. 2019; 111: 69-82
        • Bossuyt P.M.
        • Irwig L.
        • Craig J.
        • Glasziou P.
        Comparative accuracy: assessing new tests against existing diagnostic pathways.
        BMJ. 2006; 332: 1089-1092
        • Mustafa R.A.
        • Wiercioch W.
        • Cheung A.
        • Prediger B.
        • Brozek J.
        • Bossuyt P.
        • et al.
        Decision making about healthcare-related tests and diagnostic test strategies. Paper 2: a review of methodological and practical challenges.
        J Clin Epidemiol. 2017; 92: 18-28
        • Schunemann H.J.
        • Mustafa R.A.
        Decision making about healthcare-related tests and diagnostic test strategies. Paper 1: a new series on testing to improve people's health.
        J Clin Epidemiol. 2017; 92: 16-17
        • Santesso N.
        • Mustafa R.A.
        • Schunemann H.J.
        • Arbyn M.
        • Blumenthal P.D.
        • Cain J.
        • et al.
        World Health Organization Guidelines for treatment of cervical intraepithelial neoplasia 2-3 and screen-and-treat strategies to prevent cervical cancer.
        Int J Gynaecol Obstet. 2016; 132: 252-258
        • Santesso N.
        • Mustafa R.A.
        • Wiercioch W.
        • Kehar R.
        • Gandhi S.
        • Chen Y.
        • et al.
        Systematic reviews and meta-analyses of benefits and harms of cryotherapy, LEEP, and cold knife conization to treat cervical intraepithelial neoplasia.
        Int J Gynaecol Obstet. 2016; 132: 266-271
        • Oxman A.D.
        • Guyatt G.H.
        Guidelines for reading literature reviews.
        CMAJ. 1988; 138: 697-703
        • Mulrow C.
        • Linn W.D.
        • Gaul M.K.
        • Pugh J.A.
        Assessing quality of a diagnostic test evaluation.
        J Gen Intern Med. 1989; 4: 288-295
        • Fiocchi A.
        • Brozek J.
        • Schunemann H.
        • Bahna S.L.
        • von Berg A.
        • Beyer K.
        • et al.
        World allergy organization (WAO) diagnosis and rationale for action against Cow's milk allergy (DRACMA) guidelines.
        Pediatr Allergy Immunol. 2010; 21: 1-125
        • Bossuyt P.M.
        • Lijmer J.G.
        • Mol B.W.
        Randomised comparisons of medical tests: sometimes invalid, not always efficient.
        Lancet. 2000; 356: 1844-1847
        • Mueller C.
        • Scholer A.
        • Laule-Kilian K.
        • Martina B.
        • Schindler C.
        • Buser P.
        • et al.
        Use of B-type natriuretic peptide in the evaluation and management of acute dyspnea.
        N Engl J Med. 2004; 350: 647-654
        • Moe G.
        • Howlett J.
        • Januzzi J.L.
        • Zowall H.
        • Canadian Multicenter Improved Management of Patients with Congestive Heart Failure (IMPROVE-CHF) Study Investigators
        N-terminal pro-B-type natriuretic peptide testing improves the management of patients with suspected acute heart failure: primary results of the Canadian Prospective Randomized Multicenter IMPROVE-CHF study.
        Circulation. 2007; 115: 3103-3110
        • Worster A.
        • Preyra I.
        • Weaver B.
        • Haines T.
        The accuracy of noncontrast helical computed tomography versus intravenous pyelography in the diagnosis of suspected acute urolithiasis: a meta-analysis.
        Ann Emerg Med. 2002; 40: 280-286
        • Worster A.
        • Haines T.
        Does replacing intravenous pyelography with noncontrast helical computed tomography benefit patients with suspected acute urolithiasis?.
        Can Assoc Radiol. 2002; 53: 144-148
        • Sargent D.J.
        • Conley B.A.
        • Allegra C.
        • Collette L.
        Clinical trial designs for predictive marker validation in cancer treatment trials.
        J Clin Oncol. 2005; 23: 2020-2027
        • Lijmer J.G.
        • Bossuyt P.M.
        Various randomized designs can be used to evaluate medical tests.
        J Clin Epidemiol. 2009; 62: 364-373
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Atkins D.
        • Brozek J.
        • Vist G.
        • et al.
        GRADE guidelines: 2. Framing the question and deciding on important outcomes.
        J Clin Epidemiol. 2011; 64: 395-400
        • Lord S.J.
        • Irwig L.
        • Simes R.J.
        When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials?.
        Ann Intern Med. 2006; 144: 850-855
        • Evans W.K.
        • Laupacis A.
        • Gulenchyn K.Y.
        • Levin L.
        • Levine M.
        Evidence-based approach to the introduction of positron emission tomography in Ontario, Canada.
        J Clin Oncol. 2009; 27: 5607-5613
        • Maat-Kievit A.
        • Vlis M.V.
        • Zoeteweij M.
        • Losekoot M.
        • van Haeringen A.
        • Roos R.
        Paradox of a better test for Huntington's disease.
        J Neurol Neurosurg Psychiatry. 2000; 69: 579-583
        • Walker F.M.D.
        Huntington's disease.
        Semin Neurol. 2007; : 143-150
        • Almqvist E.W.
        • Brinkman R.R.
        • Wiggins S.
        • Hayden M.R.
        Psychological consequences and predictors of adverse events in the first 5 years after predictive testing for Huntington's disease.
        Clin Genet. 2003; 64: 300-309
        • Kohli M.
        • Schiller I.
        • Dendukuri N.
        • Dheda K.
        • Denkinger C.M.
        • Schumacher S.G.
        • et al.
        Xpert((R)) MTB/RIF assay for extrapulmonary tuberculosis and rifampicin resistance.
        Cochrane Database Syst Rev. 2018; 8: CD012768
        • Bossuyt P.M.
        • Reitsma J.B.
        • Bruns D.E.
        • Gatsonis C.A.
        • Glasziou P.P.
        • Irwig L.M.
        • et al.
        Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD Initiative.
        Ann Intern Med. 2003; 138: 40-44
        • Whiting P.
        • Rutjes A.W.
        • Reitsma J.B.
        • Bossuyt P.M.
        • Kleijnen J.
        The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews.
        BMC Med Res Methodol. 2003; 3: 25
        • Whiting P.F.
        • Weswood M.E.
        • Rutjes A.W.
        • Reitsma J.B.
        • Bossuyt P.N.
        • Kleijnen J.
        Evaluation of QUADAS, a tool for the quality assessment of diagnostic accuracy studies.
        BMC Med Res Methodol. 2006; 6: 9
        • Whiting P.F.
        • Rutjes A.W.S.
        • Westwood M.E.
        • Mallett S.
        • Deeks J.J.
        • Reitsma J.B.
        • et al.
        QUADAS-2: a Revised tool for the quality assessment of diagnostic accuracy studies.
        Ann Intern Med. 2011; 155: 529-536
        • Rutjes A.W.
        • Reitsma J.B.
        • Di Nisio M.
        • Smidt N.
        • van Rijn J.C.
        • Bossuyt P.M.
        Evidence of bias and variation in diagnostic accuracy studies.
        CMAJ. 2006; 174: 469-476
        • Lijmer J.G.
        • Mol B.W.
        • Heisterkamp S.
        • Bonsel G.J.
        • Prins M.H.
        • van der Meulen J.H.
        • et al.
        Empirical evidence of design-related bias in studies of diagnostic tests.
        JAMA. 1999; 282: 1061-1066
        • Steingart K.R.
        • Flores L.L.
        • Dendukuri N.
        • Schiller I.
        • Laal S.
        • Ramsay A.
        • et al.
        Commercial Serological tests for the diagnosis of active pulmonary and extrapulmonary tuberculosis: an updated systematic review and Meta-Analysis.
        PLoS Med. 2011; 8: e1001062