GRADE guidelines 17: assessing the risk of bias associated with missing participant outcome data in a body of evidence

  • Gordon H. Guyatt
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Medicine, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada
    Search for articles by this author
  • Shanil Ebrahim
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Systematic Overviews through Advancing Research Technology (SORT), Child Health Evaluative Sciences, The Hospital for Sick Children Research Institute, 555 University Ave, Toronto, ON M5G 1X8, Canada
    Search for articles by this author
  • Pablo Alonso-Coello
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Iberoamerican Cochrane Centre, CIBERESP-IIB Sant Pau, Casa de Convalescéncia, 4 th floor, C. Sant Antoni Maria Claret 171, Barcelona 08041, Spain
    Search for articles by this author
  • Bradley C. Johnston
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Systematic Overviews through Advancing Research Technology (SORT), Child Health Evaluative Sciences, The Hospital for Sick Children Research Institute, 555 University Ave, Toronto, ON M5G 1X8, Canada

    Department of Anesthesia and Pain Medicine, The Hospital for Sick Children, University of Toronto, 555 University Ave, Toronto, ON M5G 1X8, Canada

    Institute for Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, 155 College St, Toronto, ON M5T 3M7, Canada
    Search for articles by this author
  • Alexander G. Mathioudakis
    Affiliations
    Iberoamerican Cochrane Centre, CIBERESP-IIB Sant Pau, Casa de Convalescéncia, 4 th floor, C. Sant Antoni Maria Claret 171, Barcelona 08041, Spain
    Search for articles by this author
  • Matthias Briel
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Clinical Research, Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Hebelstrasse 10, Basel 4056, Switzerland
    Search for articles by this author
  • Reem A. Mustafa
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Internal Medicine, Kansas University Medical Center, 3901 Rainbow Blvd, Kansas City, KS MS3002, USA
    Search for articles by this author
  • Xin Sun
    Affiliations
    Chinese Evidence-based Medicine Center, West China Hospital, Sichuan University, Chengdu 610041, China
    Search for articles by this author
  • Stephen D. Walter
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada
    Search for articles by this author
  • Diane Heels-Ansdell
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada
    Search for articles by this author
  • Ignacio Neumann
    Affiliations
    Department of Internal Medicine, Pontificia Universidad Catolica de Chile, Av Libertador Bernardo O'Higgins 340, Santiago, Región Metropolitana, Chile
    Search for articles by this author
  • Lara A. Kahale
    Affiliations
    Department of Internal Medicine, American University of Beirut, Riad-El-Solh Beirut, Beirut 1107 2020, Lebanon
    Search for articles by this author
  • Alfonso Iorio
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Medicine, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada
    Search for articles by this author
  • Joerg Meerpohl
    Affiliations
    Cochrane Germany, Medical Center–University of Freiburg, Breisacher Strasse 153, Freiburg 79110, Germany

    Centre de Recherche Épidémiologie et Statistique Sorbonne Paris Cité–U1153, Inserm/Université Paris Descartes, Cochrane France, Hôpital Hôtel-Dieu, 1 place du Parvis Notre Dame, Paris Cedex 04 75181, France
    Search for articles by this author
  • Holger J. Schünemann
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Medicine, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada
    Search for articles by this author
  • Elie A. Akl
    Correspondence
    Corresponding author. Department of Internal Medicine, Clinical Epidemiology Unit, American University of Beirut Medical Center, P.O. Box: 11-0236, Riad-El-Solh Beirut 1107, 2020 Beirut, Lebanon.
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1200 Main St. West, Hamilton L8S 4K1, Canada

    Department of Internal Medicine, American University of Beirut, Riad-El-Solh Beirut, Beirut 1107 2020, Lebanon
    Search for articles by this author

      Abstract

      Objective

      To provide GRADE guidance for assessing risk of bias across an entire body of evidence consequent on missing data for systematic reviews of both binary and continuous outcomes.

      Study Design and Setting

      Systematic survey of published methodological research, iterative discussions, testing in systematic reviews, and feedback from the GRADE Working Group.

      Results

      Approaches begin with a primary meta-analysis using a complete case analysis followed by sensitivity meta-analyses imputing, in each study, data for those with missing data, and then pooling across studies. For binary outcomes, we suggest use of “plausible worst case” in which review authors assume that those with missing data in treatment arms have proportionally higher event rates than those followed successfully. For continuous outcomes, imputed mean values come from other studies within the systematic review and the standard deviation (SD) from the median SDs of the control arms of all studies.

      Conclusions

      If the results of the primary meta-analysis are robust to the most extreme assumptions viewed as plausible, one does not rate down certainty in the evidence for risk of bias due to missing participant outcome data. If the results prove not robust to plausible assumptions, one would rate down certainty in the evidence for risk of bias.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).
        J Clin Epidemiol. 2011; 64: 407-415
        • Balshem H.
        • Helfand M.
        • Schunemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
      1. Higgins J.P.T. Green S S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration. 2011 (Available from www.handbook.cochrane.org)
        • Akl E.A.
        • Johnston B.C.
        • Alonso-Coello P.
        • Neumann I.
        • Ebrahim S.
        • Briel M.
        • et al.
        Addressing dichotomous data for participants excluded from trial analysis: a guide for systematic reviewers.
        PLoS One. 2013; 8: e57132
        • Ebrahim S.
        • Akl E.A.
        • Mustafa R.A.
        • Sun X.
        • Walter S.D.
        • Heels-Ansdell D.
        • et al.
        Addressing continuous data for participants excluded from trial analysis: a guide for systematic reviewers.
        J Clin Epidemiol. 2013; 66: 1014-1021.e1
        • Ebrahim S.
        • Johnston B.C.
        • Akl E.A.
        • Mustafa R.A.
        • Sun X.
        • Walter S.D.
        • et al.
        Addressing continuous data measured with different instruments for participants excluded from trial analysis: a guide for systematic reviewers.
        J Clin Epidemiol. 2014; 67: 560-570
        • Akl E.A.
        • Kahale L.A.
        • Agoritsas T.
        • Brignardello-Petersen R.
        • Busse J.W.
        • Carrasco-Labra A.
        • et al.
        Handling trial participants with missing outcome data when conducting a meta-analysis: a systematic survey of proposed approaches.
        Syst Rev. 2015; 4: 98
        • Akl E.A.
        • Carrasco-Labra A.
        • Brignardello-Petersen R.
        • Neumann I.
        • Johnston B.C.
        • Sun X.
        • et al.
        Reporting, handling and assessing the risk of bias associated with missing participant data in systematic reviews: a methodological survey.
        BMJ Open. 2015; 5: e009368
        • Akl E.A.
        • Shawwa K.
        • Kahale L.A.
        • Agoritsas T.
        • Brignardello-Petersen R.
        • Busse J.W.
        • et al.
        Reporting missing participant data in randomised trials: systematic survey of the methodological literature and a proposed guide.
        BMJ Open. 2015; 5: e008431
        • Akl E.A.
        • Kahale L.A.
        • Ebrahim S.
        • Alonso-Coello P.
        • Schünemann H.J.
        • Guyatt G.H.
        Three challenges described for identifying participants with missing data in trials reports, and potential solutions suggested to systematic reviewers.
        J Clin Epidemiol. 2016; 76: 147-154
        • Abraha I.
        • Montedori A.
        Modified intention to treat reporting in randomised controlled trials: systematic review.
        BMJ. 2010; 340: c2697
        • Abraha I.
        • Cherubini A.
        • Cozzolino F.
        • De Florio R.
        • Luchetta M.L.
        • Rimland J.M.
        • et al.
        Deviation from intention to treat analysis in randomised trials and treatment effect estimates: meta-epidemiological study.
        BMJ. 2015; 350: h2445
        • Schulz K.F.
        Assessing allocation concealment and blinding in randomised controlled trials: why bother?.
        Evid Based Nurs. 2001; 4: 4-6
        • Alshurafa M.
        • Briel M.
        • Akl E.A.A.
        • Haines T.
        • Moayyedi P.
        • Gentles S.J.
        • et al.
        Inconsistent definitions for intention-to-treat in relation to missing outcome data: systematic review of the methods literature.
        PLoS One. 2012; 7: e49163
        • Higgins J.P.
        • White I.R.
        • Wood A.M.
        Imputation methods for missing outcome data in meta-analysis of clinical trials.
        Clin Trials. 2008; 5: 225-239
        • Mavridis D.
        • White I.R.
        • Higgins J.P.
        • Cipriani A.
        • Salanti G.
        Allowing for uncertainty due to missing continuous outcome data in pairwise and network meta-analysis.
        Stat Med. 2015; 34: 721-741
        • Akl E.A.
        • Briel M.
        • You J.J.
        • Sun W.
        • Johnston B.C.
        • Busse J.W.
        • et al.
        Potential impact on estimated treatment effects of information lost to follow-up in randomised controlled trials (LOST-IT): systematic review.
        BMJ. 2012; 344: e2809
        • White I.R.
        • Higgins J.P.
        • Wood A.M.
        Allowing for uncertainty due to missing data in meta-analysis–part 1: two-stage methods.
        Stat Med. 2008; 27: 711-727
        • White I.R.
        • Welton N.J.
        • Wood A.M.
        • Ades A.E.
        • Higgins J.
        Allowing for uncertainty due to missing data in meta-analysis–part 2: hierarchical models.
        Stat Med. 2008; 27: 728-745
        • Turner N.L.
        • Dias S.
        • Ades A.E.
        • Welton N.J.
        A Bayesian framework to account for uncertainty due to missing binary outcome data in pairwise meta-analysis.
        Stat Med. 2015; 34: 2062-2080
        • Geng E.H.
        • Emenyonu N.
        • Bwana M.B.
        • Glidden D.V.
        • Martin J.N.
        Sampling-based approach to determining outcomes of patients lost to follow-up in antiretroviral therapy scale-up programs in Africa.
        JAMA. 2008; 300: 506-507
        • Akl E.A.
        • Kahale L.
        • Barba M.
        • Neumann I.
        • Labedi N.
        • Terrenato I.
        • et al.
        Anticoagulation for the long-term treatment of venous thromboembolism in patients with cancer.
        Cochrane Database Syst Rev. 2014; 7: CD006650
        • Akl E.A.
        • Kahale L.
        • Sperati F.
        • Neumann I.
        • Labedi N.
        • Terrenato I.
        • et al.
        Low molecular weight heparin versus unfractionated heparin for perioperative thromboprophylaxis in patients with cancer.
        Cochrane Database Syst Rev. 2014; 6: CD009447
        • Akl E.A.
        • Kahale L.
        • Terrenato I.
        • Neumann I.
        • Yosuico V.E.
        • Barba M.
        • et al.
        Oral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation.
        Cochrane Database Syst Rev. 2014; 7: CD006466
        • Akl E.A.
        • Kahale L.A.
        • Ballout R.A.
        • Barba M.
        • Yosuico V.E.
        • van Doormaal F.F.
        • et al.
        Parenteral anticoagulation in ambulatory patients with cancer.
        Cochrane Database Syst Rev. 2014; 12: CD006652
        • Akl E.A.
        • Ramly E.P.
        • Kahale L.A.
        • Yosuico V.E.
        • Barba M.
        • Sperati F.
        • et al.
        Anticoagulation for people with cancer and central venous catheters.
        Cochrane Database Syst Rev. 2014; 10: CD006468
        • Lytvyn L.
        • Quach K.
        • Banfield L.
        • Johnston B.C.
        • Mertz D.
        Probiotics and synbiotics for the prevention of postoperative infections following abdominal surgery: a systematic review and meta-analysis of randomized controlled trials.
        J Hosp Infect. 2016; 92: 130-139
        • Johnston B.C.
        • Ma S.S.
        • Goldenberg J.Z.
        • Thorlund K.
        • Vandvik P.O.
        • Loeb M.
        • et al.
        Probiotics for the prevention of Clostridium difficile-associated diarrhea: a systematic review and meta-analysis.
        Ann Intern Med. 2012; 157: 878-888
        • Spencer F.A.
        • Sekercioglu N.
        • Prasad M.
        • Lopes L.C.
        • Guyatt G.H.
        Culprit vessel versus immediate complete revascularization in patients with ST-segment myocardial infarction-a systematic review.
        Am Heart J. 2015; 170: 1133-1139
        • Spencer F.A.
        • Lopes L.C.
        • Kennedy S.A.
        • Guyatt G.H.
        Systematic review of percutaneous closure versus medical therapy in patients with cryptogenic stroke and patent foramen ovale.
        BMJ Open. 2014; 4: e004282
        • Spencer F.A.
        • Prasad M.
        • Vandvik P.O.
        • Chetan D.
        • Zhou Q.
        • Guyatt G.H.
        Longer- versus shorter-duration dual-antiplatelet therapy after drug-eluting stent placement: a systematic review and meta-analysis.
        Ann Intern Med. 2015; 163: 118-126
      2. How to read clinical journals: V: to distinguish useful from useless or even harmful therapy.
        Can Med Assoc J. 1981; 124: 1156-1162
        • Johnston B.C.
        • Thorlund K.
        • Schunemann H.J.
        • Xie F.
        • Murad M.H.
        • Montori V.M.
        • et al.
        Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units.
        Health Qual Life Outcomes. 2010; 8: 116
        • Guyatt G.H.
        • Thorlund K.
        • Oxman A.D.
        • Walter S.D.
        • Patrick D.
        • Furukawa T.A.
        • et al.
        GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes.
        J Clin Epidemiol. 2013; 66: 173-183
        • Guyatt G.H.
        • Feeny D.H.
        • Patrick D.L.
        Measuring health-related quality of life.
        Ann Intern Med. 1993; 118: 622-629
        • Thorlund K.
        • Walter S.D.
        • Johnston B.C.
        • Furukawa T.A.
        • Guyatt G.H.
        Pooling health-related quality of life outcomes in meta-analysis-a tutorial and review of methods for enhancing interpretability.
        Res Synth Methods. 2011; 2: 188-203
        • Akl E.A.
        • Kahale L.A.
        • Agarwal A.
        • Al-Matari N.
        • Ebrahim S.
        • Alexander P.E.
        • et al.
        Impact of missing participant data for dichotomous outcomes on pooled effect estimates in systematic reviews: a protocol for a methodological study.
        Syst Rev. 2014; 3: 137