Advertisement

Customized guidance/training improved the psychometric properties of methodologically rigorous risk of bias instruments for non-randomized studies

  • Maya M. Jeyaraman
    Correspondence
    Corresponding author. Tel.:: (204) 594-5362; fax: (204) 594-5394.
    Affiliations
    George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada

    Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, R3E 0T6, Canada
    Search for articles by this author
  • Reid C. Robson
    Affiliations
    Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario, M5B 1T8, Canada
    Search for articles by this author
  • Leslie Copstein
    Affiliations
    George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada
    Search for articles by this author
  • Nameer Al-Yousif
    Affiliations
    George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada
    Search for articles by this author
  • Michelle Pollock
    Affiliations
    Institute of Health Economics, 1200-10405 Jasper Avenue, Edmonton, Alberta, T5J 3N4, Canada
    Search for articles by this author
  • Jun Xia
    Affiliations
    Division of Epidemiology and Public Health, School of Medicine, University of Nottingham Medical School, Nottingham, NG7 2UH, UK

    Nottingham Ningbo GRADE Centre, The University of Nottingham Ningbo, 199 East Taikang Road, Ningbo, China
    Search for articles by this author
  • Chakrapani Balijepalli
    Affiliations
    Pharmalytics Group, 422 Richards St, Suite 170, Vancouver, Canada
    Search for articles by this author
  • Kimberly Hofer
    Affiliations
    Evidinno Outcomes Research Inc., 1750 Davie Street, Suites 601 & 602, Vancouver, British Columbia, V6B 2Z4, Canada
    Search for articles by this author
  • Samer Mansour
    Affiliations
    Centre Hospitalier de l'Université de Montreal, 2900, boul. Édouard-Montpetit, Montréal (Québec) H3T 1J4, Canada

    Faculty of Medicine, Department of Medicine, Université de Montréal, Roger-Gaudry Building, 2900 Edouard Montpetit Blvd, Montreal, Quebec H3T 1J4, Canada

    Centre de recherche du Centre Hospitalier de l'Université de Montréal, 900 St Denis St, Montreal, Quebec H2 × 0A9, Canada
    Search for articles by this author
  • Mir S. Fazeli
    Affiliations
    Evidinno Outcomes Research Inc., 1750 Davie Street, Suites 601 & 602, Vancouver, British Columbia, V6B 2Z4, Canada
    Search for articles by this author
  • Mohammed T. Ansari
    Affiliations
    School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Room 101, 600 Peter Morand Crescent, Ottawa, Ontario, K1G 5Z3, Canada
    Search for articles by this author
  • Andrea C. Tricco
    Affiliations
    Li Ka Shing Knowledge Institute, St. Michael's Hospital, Unity Health Toronto, 209 Victoria St, Toronto, Ontario, M5B 1T8, Canada

    Epidemiology Division & Institute of Health, Management, and Policy Evaluation, Dalla Lana School of Public Health, University of Toronto, 27 King's College Circle, Toronto, Ontario, M5S 1A1, Canada

    Queen's Collaboration for Health Care Quality Joanna Briggs Institute Centre of Excellence, Queen's University, 92 Barrie Street, Room 214, Kingston, Ontario, K7L 3N6, Canada
    Search for articles by this author
  • Rasheda Rabbani
    Affiliations
    George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada

    Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, R3E 0T6, Canada
    Search for articles by this author
  • Ahmed M. Abou-Setta
    Affiliations
    George & Fay Yee Center for Healthcare Innovation, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba. R3E 0T6, Canada

    Department of Community Health Sciences, Max Rady College of Medicine, Rady Faculty of Health Sciences, University of Manitoba, Winnipeg, Manitoba, R3E 0T6, Canada
    Search for articles by this author

      Abstract

      Objectives

      To evaluate the impact of guidance and training on the inter-rater reliability (IRR), inter-consensus reliability (ICR) and evaluator burden of the Risk of Bias (RoB) in Non-randomized Studies (NRS) of Interventions (ROBINS-I) tool, and the RoB instrument for NRS of Exposures (ROB-NRSE).

      Study design and Setting

      In a before-and-after study, seven reviewers appraised the RoB using ROBINS-I (n = 44) and ROB-NRSE (n = 44), before and after guidance and training. We used Gwet's AC1 statistic to calculate IRR and ICR.

      Results

      After guidance and training, the IRR and ICR of the overall bias domain of ROBINS-I and ROB-NRSE improved significantly; with many individual domains showing either a significant (IRR and ICR of ROB-NRSE; ICR of ROBINS-I), or nonsignificant improvement (IRR of ROBINS-I). Evaluator burden significantly decreased after guidance and training for ROBINS-I, whereas for ROB-NRSE there was a slight nonsignificant increase.

      Conclusion

      Overall, there was benefit for guidance and training for both tools. We highly recommend guidance and training to reviewers prior to RoB assessments and that future research investigate aspects of guidance and training that are most effective.

      Keywords

      Abbreviations:

      CI (Confidence interval), ICR (Inter-consensus reliability), IRR (Inter-rater reliability), NOS (Newcastle-Ottawa Scale), NRS (Non-randomized studies), NRSE (Non-randomized studies of exposures), NRSI (Non-randomized studies of interventions), RCT (Randomized controlled trials), ROB (Risk of Bias), ROBINS-I (Risk of Bias in Non-randomized Studies of Interventions), ROB-NRSE (Risk of Bias instrument for Non-randomized Studies of Exposures)
      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

      1. Schünemann H, Brożek J, Guyatt G, Oxman A, eds. GRADE handbook for grading quality of evidence and strength of recommendations. Updated October 2013. The GRADE Working Group, 2013. Available fromguidelinedevelopment.org/handbook.

      2. Higgins JPT Green S Cochrane Handbook for Systematic Reviews of Interventions. The Cochrane Collaboration, 2011 (Version 5.1.0 [updated March 2011]Available from)
        • Sterne JA
        • Hernan MA
        • Reeves BC
        • Savovic J
        • Berkman MD
        • Viswanathan M
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        Bmj. 2016; 355: i4919
        • Morgan RL
        • Thayer KA
        • Santesso N
        • Holloway AC
        • Blain R
        • Eftim SE
        • et al.
        Evaluation of the risk of bias in non-randomized studies of interventions (ROBINS-I) and the ‘target experiment’ concept in studies of exposures: Rationale and preliminary instrument development.
        Environ Int. 2018; 120: 382-387
      3. The Risk of Bias Instrument for Non-randomized Studies of Exposures. Available at: http://heigrade.mcmaster.ca/guideline-development/rob-instrument-for-nrs-of-exposures.

        • Schunemann HJ
        • Cuello C
        • Akl EA
        • Mustafa RA
        • Meerpohl JJ
        • Thayer K
        • et al.
        GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence.
        J Clin Epidemiol. 2018; 111: 105-114
        • Jeyaraman MM
        • Al-Yousif N
        • Robson RC
        • Copstein L
        • Balijepalli C
        • Hofer K
        • et al.
        Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol.
        Syst Rev. 2020; 9: 32
        • Jeyaraman MM
        • Rabbani R
        • Al-Yousif N
        • Robson RC
        • Copstein L
        • Xia J
        • et al.
        Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study.
        Syst Rev. 2020; 9: 12
        • Jeyaraman MM
        • Rabbani R
        • Copstein L
        • Robson RC
        • Al-Yousif N
        • Pollock M
        • et al.
        Methodologically rigorous risk of bias tools for non-randomized studies had low reliability and high evaluator burden.
        J Clin Epidemiol. 2020; 128: 140-147
        • Losilla JM
        • Oliveras I
        • Marin-Garcia JA
        • Vives J.
        Three risk of bias tools lead to opposite conclusions in observational research synthesis.
        J Clin Epidemiol. 2018; 101: 61-72
        • Minozzi S
        • Cinquini M
        • Gianola S
        • Castellini G
        • Gerardi C
        • Banzi R.
        Risk of bias in nonrandomized studies of interventions showed low inter-rater reliability and challenges in its application.
        J Clin Epidemiol. 2019; 112: 28-35
        • Farrah K
        • Young K
        • Tunis MC
        • Zhao L.
        Risk of bias tools in systematic reviews of health interventions: an analysis of PROSPERO-registered protocols.
        Syst Rev. 2019; 8: 280
      4. Inter-rater reliability and concurrent validity of ROBINS-I: protocol for a cross-sectional study. Open Science Framework (OSF) pre-registration - https://osf.io/k5gnc. 2020.

      5. Inter-rater reliability and validity of risk of bias instrument for non-randomized studies of exposures: a study protocol. Open Science Framework (OSF) pre-registration - https://osf.io/duv7x. 2020.

      6. Sterne JAC, Higgins JPT, Elbers RG, Reeves BC and the development group for ROBINS-I. Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance, updated 12 October 2016. Available from http://www.riskofbias.info [accessed {September 2018}].

        • Morgan RL
        • Thayer KA
        • Santesso N
        • Holloway AC
        • Blain R
        • Eftim SE
        • et al.
        A risk of bias instrument for non-randomized studies of exposures: a users’ guide to its application in the context of GRADE.
        Environ Int. 2019; 122: 168-184
      7. Sterne JAC, Higgins JPT, Elbers RG, Reeves BC and the development group for ROBINS-I. Risk of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance, updated 12 October 2016. Available from http://www.riskofbias.info [accessed September, 2018].

        • Cook DA
        • Beckman TJ.
        Current concepts in validity and reliability for psychometric instruments: theory and application.
        Am J Med. 2006; 119 (166-116): e167
        • Downing SM.
        Validity: on meaningful interpretation of assessment data.
        Med Educ. 2003; 37: 830-837
        • Gwet KL.
        Computing inter-rater reliability and its variance in the presence of high agreement.
        Br J Math Stat Psychol. 2008; 61: 29-48
        • Landis JR
        • Koch GG.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174