GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)

      Abstract

      In the GRADE approach, randomized trials start as high-quality evidence and observational studies as low-quality evidence, but both can be rated down if most of the relevant evidence comes from studies that suffer from a high risk of bias. Well-established limitations of randomized trials include failure to conceal allocation, failure to blind, loss to follow-up, and failure to appropriately consider the intention-to-treat principle. More recently recognized limitations include stopping early for apparent benefit and selective reporting of outcomes according to the results. Key limitations of observational studies include use of inappropriate controls and failure to adequately adjust for prognostic imbalance. Risk of bias may vary across outcomes (e.g., loss to follow-up may be far less for all-cause mortality than for quality of life), a consideration that many systematic reviews ignore. In deciding whether to rate down for risk of bias—whether for randomized trials or observational studies—authors should not take an approach that averages across studies. Rather, for any individual outcome, when there are some studies with a high risk, and some with a low risk of bias, they should consider including only the studies with a lower risk of bias.

      Keywords

      To read this article in full you will need to make a payment

      References

      1. The users’ guides to the medical literature: a manual for evidence-based clinical practice.
        in: Guyatt G. Rennie D. Meade M. Cook D. 2nd ed. McGraw-Hill, New York, NY2008
        • Higgins J.P.
        • Altman D.
        Assessing the risk of bias in included studies.
        in: Higgins J. Green S. Cochrane handbook for systematic reviews of interventions 5.0.1. John Wiley & Sons, Chichester, UK2008
        • Vedula S.S.
        • Bero L.
        • Scherer R.W.
        • Dickersin K.
        Outcome reporting in industry-sponsored trials of gabapentin for off-label use.
        N Engl J Med. 2009; 361: 1963-1971
        • Mathieu S.
        • Boutron I.
        • Moher D.
        • Altman D.G.
        • Ravaud P.
        Comparison of registered and published primary outcomes in randomized controlled trials.
        JAMA. 2009; 302: 977-984
        • Wood L.
        • Egger M.
        • Gluud L.L.
        • Schulz K.F.
        • Jüni P.
        • Altman D.G.
        • et al.
        Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study.
        BMJ. 2008; 336: 601-605
        • Pocock S.J.
        When (not) to stop a clinical trial for benefit.
        JAMA. 2005; 294: 2228-2230
        • Pocock S.J.
        • Hughes M.D.
        Practical problems in interim analyses, with particular regard to estimation.
        Control Clin Trials. 1989; 10: 209S-221S
        • Montori V.M.
        • Devereaux P.J.
        • Adhikari N.K.
        • Burns K.E.
        • Eggert C.H.
        • Briel M.
        • et al.
        Randomized trials stopped early for benefit: a systematic review.
        JAMA. 2005; 294: 2203-2209
        • Bassler D.
        • Briel M.
        • Montori V.M.
        • Lane M.
        • Glasziou P.
        • Zhou Q.
        • et al.
        Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis.
        JAMA. 2010; 303: 1180-1187
        • Furukawa T.A.
        • Watanabe N.
        • Omori I.M.
        • Montori V.M.
        • Guyatt G.H.
        Association between unreported outcomes and effect size estimates in Cochrane meta-analyses.
        JAMA. 2007; 297: 468-470
        • Chan A.W.
        • Altman D.G.
        Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors.
        BMJ. 2005; 330: 753
        • Chan A.W.
        • Hróbjartsson A.
        • Haahr M.T.
        • Gøtzsche P.C.
        • Altman D.G.
        Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles.
        JAMA. 2004; 291: 2457-2465
        • Chan A.W.
        • Krleza-Jerić K.
        • Schmid I.
        • Altman D.G.
        Outcome reporting bias in randomized trials funded by the Canadian Institutes of Health Research.
        CMAJ. 2004; 171: 735-740
        • Bolona E.R.
        • Uraga M.V.
        • Haddad R.M.
        • Tracz M.J.
        • Sideras K.
        • Kennedy C.C.
        • et al.
        Testosterone use in men with sexual dysfunction: a systematic review and meta-analysis of randomized placebo-controlled trials.
        Mayo Clin Proc. 2007; 82: 20-28
        • Sinha M.
        • Montori V.M.
        Reporting bias and other biases affecting systematic reviews and meta-analyses: a methodological commentary.
        Expert Rev Pharmacoecon Outcomes Res. 2006; 6: 603-611
        • Deeks J.J.
        • Dinnes J.
        • D’Amico R.
        • Sowden A.J.
        • Sakarovitch C.
        • Song F.
        • et al.
        Evaluating non-randomised intervention studies.
        Health Technol Assess. 2003; 7 (1–173): iii-x
        • West S.
        • King V.
        • Carey T.S.
        • Lohr K.N.
        • Mckoy N.
        • Sutton S.F.
        • et al.
        Systems to rate the strength of scientific evidence [evidence report/technology assessment no 47]. AHRQ Publication No 02–E016.
        Agency for Healthcare Research and Quality, Rockville, MD2002
      2. Proposed Evaluation Tools for COMPUS: assessment, November 29, 2005.
        Canadian Coordinating Office for Health Technology, Ottawa, Canada2005
        • Sanderson S.
        • Tatt I.D.
        • Higgins J.P.
        Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography.
        Int J Epidemiol. 2007; 36: 666-676
        • Greer I.A.
        • Nelson-Piercy C.
        Low-molecular-weight heparins for thromboprophylaxis and treatment of venous thromboembolism in pregnancy: a systematic review of safety and efficacy.
        Blood. 2005; 106: 401-407
        • Sanson B.J.
        • Lensing A.W.
        • Prins M.H.
        • Ginsberg J.S.
        • Barkagan Z.S.
        • Lavenne-Pardonge E.
        • et al.
        Safety of low-molecular-weight heparin in pregnancy: a systematic review.
        Thromb Haemost. 1999; 81: 668-672
        • Bowker S.L.
        • Majumdar S.R.
        • Veugelers P.
        • Johnson J.A.
        Increased cancer-related mortality for patients with type 2 diabetes who use sulfonylureas or insulin.
        Diabetes Care. 2006; 29: 254-258
        • Bracken M.B.
        • Shepard M.J.
        • Collins Jr., W.F.
        • Holford T.R.
        • Baskin D.S.
        • Eisenberg H.M.
        • et al.
        Methylprednisolone or naloxone treatment after acute spinal cord injury: 1-year follow-up data. Results of the second National Acute Spinal Cord Injury Study.
        J Neurosurg. 1992; 76: 23-31
        • Otani K.
        • Abe H.
        • Kadoya S.
        Beneficial effect of methylprednisolone sodium succinate in the treatment of acute spinal cord injury.
        Sekitsui Sekizui. 1994; 7: 633-647
        • Pointillart V.
        • Petitjean M.E.
        • Wiart L.
        • Vital J.M.
        • Lassié P.
        • Thicoipé M.
        • et al.
        Pharmacological therapy of spinal cord injury during the acute phase.
        Spinal Cord. 2000; 38: 71-76
        • Brophy J.M.
        • Joseph L.
        • Rouleau J.L.
        Beta-blockers in congestive heart failure. A Bayesian meta-analysis.
        Ann Intern Med. 2001; 134: 550-560
        • McIntosh H.
        • Jones K.
        Chloroquine or amodiaquine combined with sulfadoxine-pyrimethamine for treating uncomplicated malaria.
        Cochrane Database Syst Rev. 2005; (CD000386)
        • Gibson J.
        • Waddell G.
        Surgical interventions for lumbar disc prolapse.
        Cochrane Database Syst Rev. 2007; (CD001350)https://doi.org/10.1002/14651858
        • Dobre D.
        • van Jaarsveld C.H.
        • deJongste M.J.
        • Haaijer Ruskamp F.M.
        • Ranchor A.V.
        The effect of beta-blocker therapy on quality of life in heart failure patients: a systematic review and meta-analysis.
        Pharmacoepidemiol Drug Saf. 2007; 16: 152-159
        • Moher D.
        • Jadad A.R.
        • Nichol G.
        • Penman M.
        • Tugwell P.
        • Walsh S.
        Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists.
        Control Clin Trials. 1995; 16: 62-73
        • Schulz K.F.
        • Chalmers I.
        • Hayes R.J.
        • Altman D.G.
        Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials.
        JAMA. 1995; 273: 408-412
        • Emerson J.D.
        • Burdick E.
        • Hoaglin D.C.
        • Mosteller F.
        • Chalmers T.C.
        An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials.
        Control Clin Trials. 1990; 11: 339-352
        • Juni P.
        • Witschi A.
        • Bloch R.
        • Egger M.
        The hazards of scoring the quality of clinical trials for meta-analysis.
        JAMA. 1999; 282: 1054-1060
        • Devereaux P.J.
        • Choi P.T.
        • El-Dika S.
        • Bhandari M.
        • Montori V.M.
        • Schünemann H.J.
        • et al.
        An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods.
        J Clin Epidemiol. 2004; 57: 1232-1236
        • Soares H.P.
        • Daniels S.
        • Kumar A.
        • Clarke M.
        • Scott C.
        • Swann S.
        • Djulbegovic B.
        • et al.
        Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group.
        BMJ. 2004; 328: 22-24
        • Alonso-Coello P.
        • Zhou Q.
        • Martinez-Zapata M.J.
        • Mills E.
        • Heels-Ansdell D.
        • Johanson J.F.
        • et al.
        Meta-analysis of flavonoids for the treatment of haemorrhoids.
        Br J Surg. 2006; 93: 909-920
        • Clarke M.
        • Hopewell S.
        • Juszczak E.
        • Eisinga A.
        • Kjeldstrøm M.
        Compression stockings for preventing deep vein thrombosis in airline passengers.
        Cochrane Database Syst Rev. 2007; (CD004002)
        • Geerts W.H.
        • Bergqvist D.
        • Pineo G.F.
        • Heit J.A.
        • Samama C.M.
        • Lassen M.R.
        • et al.
        Prevention of venous thromboembolism: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition).
        Chest. 2008; 133: 381S-453S