Advertisement

GRADE notes: How to use GRADE when there is “no” evidence? A case study of the expert evidence approach

  • Reem A. Mustafa
    Correspondence
    Corresponding author. Tel.: +1 913 588 6048.
    Affiliations
    Department of Medicine, Division of Nephrology and Hypertension, University of Kansas Medical Center, 3901 Rainbow Blvd, MS3002, Kansas City, KS 66160, USA

    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
  • Carlos A. Cuello Garcia
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada

    Departments of Pediatrics and Critical Care, McMaster University, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
  • Meha Bhatt
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
  • John J. Riva
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada

    Department of Family Medicine, McMaster University, David Braley Health Sciences Centre, 100 Main Street West, 6th Floor, Hamilton, ON L8P 1H6, Canada
    Search for articles by this author
  • Sara Vesely
    Affiliations
    Department of Biostatistics and Epidemiology, Hudson College of Public Health, University of Oklahoma Health Sciences Center, 801 NE 13th, Oklahoma City, OK 73104, USA
    Search for articles by this author
  • Wojtek Wiercioch
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
  • Robby Nieuwlaat
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
  • Payal Patel
    Affiliations
    Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, Department of Medicine, Emory University, Atlanta, GA 30322, USA
    Search for articles by this author
  • Sheila Hanson
    Affiliations
    Department of Pediatrics, Medical College of Wisconsin and Critical Care Section, Children's Hospital of Wisconsin, Milwaukee, WI
    Search for articles by this author
  • Fiona Newall
    Affiliations
    Department of Nursing Research, Royal Children's Hospital, University of Melbourne, Murdoch Children's Research Institute, Melbourne, VIC, Australia

    Department of Clinical Haematology, Royal Children's Hospital, University of Melbourne, Murdoch Children's Research Institute, Melbourne, VIC, Australia
    Search for articles by this author
  • John Wiernikowski
    Affiliations
    Division of Hematology/Oncology, Department of Pediatrics, McMaster Children's Hospital, McMaster University, Hamilton, ON, Canada
    Search for articles by this author
  • Paul Monagle
    Affiliations
    Department of Clinical Haematology, Royal Children's Hospital, University of Melbourne, Murdoch Children's Research Institute, Melbourne, VIC, Australia
    Search for articles by this author
  • Holger J. Schünemann
    Affiliations
    Department of Health Research Methods, Evidence and Impact, McMaster University, 1280 Main St West, Hamilton, ON L8S 4L8, Canada
    Search for articles by this author
Open AccessPublished:March 03, 2021DOI:https://doi.org/10.1016/j.jclinepi.2021.02.026

      Abstract

      Objectives

      One essential requirement of trustworthy guidelines is that they should be based on systematic reviews of the best available evidence. The GRADE Working Group has provided guidance for evaluating the certainty of evidence based on several domains. However, for many clinical questions, published evidence may be limited, too indirect or simply not exist. In this brief report (GRADE notes), we describe our method of developing evidence-based recommendations when publisheddirect evidence was lacking.

      Study Design and Setting

      When direct published literature was absent, an expert evidence survey was administered to panel members about their unpublished observations and case series. Focus was on collecting data about cases and outcome, not panel opinions.

      Results

      Out of 26 questions prioritized by the panel for pediatric venous thromboembolism, 12 had no, very limited, or very low certainty of evidence to inform them. The panel survey was administered for these questions.

      Conclusions

      Areas of sparse evidence often reflect key questions that are critical to address in clinical practice guidelines due to the uncertainty among health care providers. The expert evidence approach used in this study is one method for panels totransparently deal with the lack of published evidence to directly inform recommendations.

      Graphical Abstract

      Keywords

      What is new?

        Key findings

      • Guideline developers can apply GRADE and use a structured approach of summarizing collective expert experience when published evidence is limited or non-existing

        What this adds to what is known

      • The GRADE Working Group has provided guidance for evaluating the certainty of evidence based on several domains. However, guidance about developing evidence-based recommendations when published direct evidence is lacking is needed. In this paper, we provide a case example as a suggested solution following the principles of collecting expert evidence.

        What is the implication/what should change now

      • Using an expert evidence approach provides a potential solution to answer clinical dilemmas that are critical to address in clinical practice guidelines when published evidence is limited or non-existing.
      • Other groups facing the dilemma of limited or lacking evidence should aim to identify methods that will allow for a structured and systematic summary of the collective experience of panelists.

      1. Background

      Clinical practice guideline recommendations are systematically developed statements intended to assist clinicians and patients in making decisions about appropriate diagnosis and management strategies in specific circumstances [
      ,
      . Essential requirements of trustworthy guidelines include that they should be based on systematic reviews of the best available evidence [
      ,
      • Qaseem A.
      • Forland F.
      • Macbeth F.
      • Ollenschläger G.
      • Phillips S
      • van der Wees P
      • et al.
      Guidelines international network: toward international standards for clinical practice guidelines.
      and that they should incorporate an assessment of the quality or certainty of evidence [
      • Hultcrantz M.
      • Rind D.
      • Akl E.A.
      • Treweek S.
      • Mustafa R.A.
      • Iorio A.
      • et al.
      The GRADE Working Group clarifies the construct of certainty of evidence.
      ,
      • Schunemann H.J..
      Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision?.
      . The GRADE Working Group has provided guidance for evaluating the certainty of evidence based on multiple domains [
      • Balshem H.
      • Helfand M.
      • Schunemann H.J.
      • Oxman A.D.
      • Kunz R.
      • Brozek J.
      • et al.
      GRADE guidelines: 3. Rating the quality of evidence.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Kunz R.
      • Brozek J.
      • Alonso-Coello P.
      • Rind D.
      • et al.
      GRADE guidelines 6. Rating the quality of evidence—imprecision.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Kunz R.
      • Woodcock J.
      • Brozek J.
      • Helfand M.
      • et al.
      GRADE guidelines: 8. Rating the quality of evidence–indirectness.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Kunz R.
      • Woodcock J.
      • Brozek J.
      • Helfand M.
      • et al.
      GRADE guidelines: 7. Rating the quality of evidence—inconsistency.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Montori V.
      • Vist G.
      • Kunz R.
      • Brozek J.
      • et al.
      GRADE guidelines: 5. Rating the quality of evidence—publication bias.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Sultan S.
      • Glasziou P.
      • Akl E.A.
      • Alonso-Coello P.
      • et al.
      GRADE guidelines: 9. Rating up the quality of evidence.
      ,
      • Guyatt G.H.
      • Oxman A.D.
      • Vist G.
      • Kunz R.
      • Brozek J.
      • Alonso-Coello P.
      • et al.
      GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).
      ,
      • Schunemann H.J.
      • Best D.
      • Vist G.
      • Oxman A.D.
      • Group G.W.
      Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
      ]. However, for many clinical questions, published evidence may be limited, too indirect or not exist, and these often are the clinical dilemmas that are critical to address in guidelines due to the uncertainty among clinicians about how best to proceed. This uncertainty is due to the lack of clear evidence to support one management strategy versus another or due to longstanding clinical heuristics that have remained unchallenged. In this brief report (GRADE notes), we describe our method of developing evidence-based recommendations when published evidence was lacking using a collective experience approach based on our prior expert evidence framework [
      • Schunemann H.J.
      • Zhang Y.
      • Oxman A.D.
      Expert evidence in guidelines G. Distinguishing opinion from evidence in guidelines.
      ].

      2. Methods

      As part of the American Society of Hematology panel for development of clinical practice guidelines to address treatment of venous thromboembolism in the pediatric population, an extensive search of published literature, including gray literature was performed [
      • Monagle P.
      • Cuello C.
      • Augustine C.
      • Bonduel M.
      • Brandão L.R.
      • Capman T.
      • et al.
      American Society of Hematology 2018 guidelines for management of venous thromboembolism: treatment of pediatric venous thromboembolism.
      ]. The evidence synthesis team was unable to identify published direct or indirect evidence for several questions or for critical outcomes. At that point, the panel had the option of not issuing any recommendations for these questions; issuing recommendations based on expert opinion without specifying the evidence; or producing primary data that could inform these recommendations. To solve the issue of lack of evidence, we administered an expert evidence survey that asks panel members about their unpublished observations and case series [
      • Schunemann H.J.
      • Zhang Y.
      • Oxman A.D.
      Expert evidence in guidelines G. Distinguishing opinion from evidence in guidelines.
      ]. The interdisciplinary guideline panel included 14 members from different specialties and countries. At least 50% of panel members had to have no financial conflict of interest regarding the recommendations being addressed. We collected information about the number of cases that the panel clinicians had managed in their career with different interventions and treatments, and outcomes observed when using these management pathways. We emphasized the focus on collecting data about cases and outcomes, rather than panel opinions that are not linked to evidence. We did not require that panel members review patients’ charts, and many completed the survey based on their memory although this would be ideal if feasible. Appendix 1 summarizes our complete survey with responses. We collected the survey results anonymously online before the guideline panel meeting. We presented the results of the survey either on its own when published evidence was completely lacking or in association with published data when the certainty of evidence was very low (e.g. evidence was extremely indirect to the population or intervention of interest). The panel then considered this evidence in their deliberations to formulate a recommendation.

      3. Results

      Out of 26 questions prioritized by the guideline panel listed in Appendix 2, 12 had no, very limited, or very low certainty evidence to inform them. Table 1 summarizes the results of the survey questions asking expert panelists to provide an approximate number of pediatric patients with specific conditions they have managed throughout their career. In Table 2, we summarize the number of pediatric patients identified through the literature search from published case reports and case series compared to the approximate number of cases that were treated by the experts on this guideline panel. Ultimately, the panel formulated 26 recommendations based on very low certainty of evidence.
      Table 1Focused expert panel survey results regarding number of patient cases by diagnosis (n = 13)
      QuestionResponses by panel memberTotalAverage (SD)
      12345678910111213
      How many years have you been practicing in your field?1718201120NR2011301025NRNR-18 (6)
      How many pediatric patients with symptomatic DVT or PE have you managed?150018010010010001000100030091411005008008009294715 (440)
      How many pediatric patients with asymptomatic DVT or PE have you managed?500755050300500200100434001000NRNR3218293 (295)
      How many pediatric patients with central venous line related thrombosis have you managed?1000125100801000750500250437151000NRNR5563506 (401)
      How many pediatric patients with central venous access devise related superficial vein thrombosis have you managed?300155025040510010250100NRNR92284 (101)
      How many neonate patients with right atrial thrombosis have you managed?3015105302010002025200NRNR45541 (59)
      How many pediatric patients with portal vein thrombosis have you managed during your years of practice?401503152702553030070800NR1609134 (233)
      How many pediatric patients with purpura fulminans due to congenital protein C deficiency have you managed in your career?01203000010110NR272 (4)
      NA, not applicable; NR, no response.
      *Some panel members provided an approximate percentage or range. If they reported a range, the mid value was used to determine the number of cases.
      Table 2Number of pediatric patients from published case reports and case series compared to the number of cases treated by expert panel by guideline question
      Guideline questionNumber of pediatric cases from published studiesNumber of pediatric cases treated by expert panel
      Q1 Symptomatic venous thromboembolism16379294
      Q2 Asymptomatic venous thromboembolism2983218
      Q9, Q10, Q11, Q12 Central venous line related thrombosis175563
      Q16 Superficial vein thrombosis0922
      Q17 Right atrial thrombosis99455
      Q21 Portal vein thrombosis6331609
      Q24, Q25, Q26 Congenital purpura fulminans7327

      4. Discussion

      While expert opinion is often based on very low certainty of evidence [
      • Ponce O.J.
      • Alvarez-Villalobos N.
      • Shah R.
      • Mohammed K.
      • Morgan R.L.
      • Sultan S.
      • et al.
      What does expert opinion in guidelines mean? A meta-epidemiological study.
      ,
      • Tricoci P.
      • Allen J.M.
      • Kramer J.M.
      • Califf R.M.
      • Smith Jr S.C.
      Scientific evidence underlying the ACC/AHA clinical practice guidelines.
      , the expertise and collective experience of guideline panels is invaluable. To mitigate inherent personal biases, efforts to collect panel members’ cumulative experience before issuing a recommendation can be beneficial. In this evidence synthesis effort, we were able to summarize evidence for an estimated number of over 12,000 individual pediatric cases, when what is published in the literature included only 1,900 cases. While there is a real potential for inflation in panel survey results, this could also reflect a possibility of underreporting and lack of published literature on this subject. In this guideline, the panel had the option of summarizing the evidence but not issuing any recommendations when the evidence was too uncertain. The panel believed that this approach would not be useful to clinicians and could cause harm to patients. The panel also realized that higher certainty published evidence may not be available in the near future. Given the complexity of these patients, the panel attempted to provide clinicians with the best suggested management plan based on the best available evidence, even if this evidence was not published.
      Guidelines developed using consensus-based techniques solely on the bases of expert opinion in the absence of published or unpublished evidence, such as the modified Delphi technique, are referred to as consensus-based guidelines—implying a consensus of opinion [
      • Brown B.
      Delphi process: a methodology used for elicitation of opinions of experts.
      ]. There is a common misunderstanding that evidence-based approaches to guidelines do not require consensus. Indeed, most guidelines require consensus, but a key difference is how evidence is used and how it is provided. We previously laid out our framework for using expert evidence which we applied in this study successfully.

      5. Strength and limitations

      This case study has multiple strengths. While many in the guideline development field address the importance and practical implications of considering experts’ experience, few have used explicit expert evidence [
      • Al-Hameed F.
      • Al-Dorzi H.M.
      • Shamy A.
      • et al.
      The Saudi clinical practice guideline for the diagnosis of the first deep venous thrombosis of the lower extremity.
      ,
      • Al-Mandeel H.M.
      • Sagr E.
      • Sait K.
      • et al.
      Clinical practice guidelines on the screening and treatment of precancerous lesions for cervical cancer prevention in Saudi Arabia.
      ,
      • Pai M.
      • Santesso N.
      • Yeung C.H.
      • Lane S.J.
      • Schunemann H.J.
      • Iorio A.
      Methodology for the development of the NHF-McMaster Guideline on care models for haemophilia management.
      ]. The expert evidence approach used in this study is one method for panels to transparently deal with the lack of published evidence to directly inform recommendations. This approach appeared beneficial to avoid the loudest voice in the room unduly influencing the recommendations. The process was also informative when the panel realized the degree of variability that exists in their practice and outcomes observed. Following this approach, thinking shifts from considerations about being not explicit to being explicit and provides opportunity for a more structured consideration of the collective data. In effect, the panel is making an evidence-based judgment (based on very low certainty evidence, i.e. the results of the survey combined with any very limited published data) using the same collective evidence, rather than individual opinions that may not be based on a common understanding, which we believe limits individual biases. Finally, this survey was developed and administered by the systematic review team based only on the areas where the evidence was lacking. The goal was to find solutions when published literature was lacking, too indirect or did not exist, rather than replace the importance of published and peer-reviewed literature using our expert evidence approach.
      This approach has multiple limitations. Any survey is inherently limited, as it may be biased by issues related to recall or perception [
      • Coughlin S.S.
      Recall bias in epidemiologic studies.
      ]. However, considering the lack of higher quality evidence, this summary of best available evidence, albeit of very low-certainty and with many limitations, is better than basing decisions on the isolated, individual experience. In this example, we did not request panelists to review patient's data as this would not have been feasible given the timeline for this project. Therefore, the reported number of cases was based on the clinician's best estimate. In areas of clinical practice where published evidence remains very limited, it is important to involve experts with years of expertise in the area, as was the case with this panel. The guideline panel also reflects practice in a variety of geographically and academically diverse centers. One can also argue that the survey is limited by restricting it to clinical experts on one panel. Other guideline panels have expanded survey to experts outside the panel which may increase generalizability or conduct chart reviews; however, this needs to be balanced with feasibility, potential biases, and conflicts of interest [
      • Mustafa R.A.
      • Zimmerman D.
      • Rioux J.P.
      • Lindsay R.
      • Pierratos A.
      • Nesrallah G.E.
      Vascular access for intensive maintenance hemodialysis: a systematic review for a Canadian Society of Nephrology clinical practice guideline.
      ,
      • Nesrallah G.E.
      • Mustafa R.A.
      • MacRae J.
      • Pauly R.P.
      • Perkins D.N.
      • Gangji A.
      • et al.
      Canadian Society of Nephrology guidelines for the management of patients with ESRD treated with intensive hemodialysis.
      . Despite the potential for inflation in survey results, the survey also confirmed to panel members the scarcity of the data published in their field. Panelists realized the need for comprehensive documentation of consecutive cases and well-structured registries for pediatric VTE cases. With this GRADE note, we invite others to use this example of the expert evidence approach to guide the process of collecting unpublished data when needed for guideline development.

      Financial disclosure

      The case example used in this article is based on work that informed the American Society of Hematology (ASH) Pediatrics Venous Thromboembolism Guidelines. The work for the guideline was funded by ASH.

      Authors’ contributions

      R.A.M., C.A.C., and H.J.S. conceived and designed the experiments. R.A.M., C.A.C., and M.B. designed the questionnaire and collected data. R.A.M., C.A.C., M.B., J.R., S.V., W.W., R.N., P.P., S.H., F.N., J.W., P.M., and H.J.S. critically appraised the data, analysis and interpretation of the data. R.A.M. wrote the first draft of the paper. All authors critically reviewed and edited the paper. C.A.C. and P.P. designed and completed the visual abstract.

      Appendix. Supplementary materials

      References

      1. Field M.J. Lohr K.N. Clinical practice guidelines: directions for a new program. Washington, DC, National Academies Press, 1990
      2. Graham R. Mancher M. Miller Wolman D. Greenfield S. Steinberg E. Clinical practice guidelines we can trust. Washington, DC, National Academies Press, 2011
        • Qaseem A.
        • Forland F.
        • Macbeth F.
        • Ollenschläger G.
        • Phillips S
        • van der Wees P
        • et al.
        Guidelines international network: toward international standards for clinical practice guidelines.
        Ann Intern Med. 2012; 156: 525-531
        • Hultcrantz M.
        • Rind D.
        • Akl E.A.
        • Treweek S.
        • Mustafa R.A.
        • Iorio A.
        • et al.
        The GRADE Working Group clarifies the construct of certainty of evidence.
        J Clin Epidemiol. 2017; 87: 4-13
        • Schunemann H.J..
        Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision?.
        J Clin Epidemiol. 2016; 75: 6-15
        • Balshem H.
        • Helfand M.
        • Schunemann H.J.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 3. Rating the quality of evidence.
        J Clin Epidemiol. 2011; 64: 401-406
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • Rind D.
        • et al.
        GRADE guidelines 6. Rating the quality of evidence—imprecision.
        J Clin Epidemiol. 2011; 64: 1283-1293
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Woodcock J.
        • Brozek J.
        • Helfand M.
        • et al.
        GRADE guidelines: 8. Rating the quality of evidence–indirectness.
        J Clin Epidemiol. 2011; 64: 1303-1310
        • Guyatt G.H.
        • Oxman A.D.
        • Kunz R.
        • Woodcock J.
        • Brozek J.
        • Helfand M.
        • et al.
        GRADE guidelines: 7. Rating the quality of evidence—inconsistency.
        J Clin Epidemiol. 2011; 64: 1294-1302
        • Guyatt G.H.
        • Oxman A.D.
        • Montori V.
        • Vist G.
        • Kunz R.
        • Brozek J.
        • et al.
        GRADE guidelines: 5. Rating the quality of evidence—publication bias.
        J Clin Epidemiol. 2011; 64: 1277-1282
        • Guyatt G.H.
        • Oxman A.D.
        • Sultan S.
        • Glasziou P.
        • Akl E.A.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 9. Rating up the quality of evidence.
        J Clin Epidemiol. 2011; 64: 1311-1316
        • Guyatt G.H.
        • Oxman A.D.
        • Vist G.
        • Kunz R.
        • Brozek J.
        • Alonso-Coello P.
        • et al.
        GRADE guidelines: 4. Rating the quality of evidence–study limitations (risk of bias).
        J Clin Epidemiol. 2011; 64: 407-415
        • Schunemann H.J.
        • Best D.
        • Vist G.
        • Oxman A.D.
        • Group G.W.
        Letters, numbers, symbols and words: how to communicate grades of evidence and recommendations.
        CMAJ. 2003; 169: 677-680
        • Schunemann H.J.
        • Zhang Y.
        • Oxman A.D.
        Expert evidence in guidelines G. Distinguishing opinion from evidence in guidelines.
        BMJ. 2019; 366: l4606
        • Monagle P.
        • Cuello C.
        • Augustine C.
        • Bonduel M.
        • Brandão L.R.
        • Capman T.
        • et al.
        American Society of Hematology 2018 guidelines for management of venous thromboembolism: treatment of pediatric venous thromboembolism.
        Blood Adv. 2018; 2: 3292-3316
        • Ponce O.J.
        • Alvarez-Villalobos N.
        • Shah R.
        • Mohammed K.
        • Morgan R.L.
        • Sultan S.
        • et al.
        What does expert opinion in guidelines mean? A meta-epidemiological study.
        Evid Based Med. 2017; 22: 164-169
        • Tricoci P.
        • Allen J.M.
        • Kramer J.M.
        • Califf R.M.
        • Smith Jr S.C.
        Scientific evidence underlying the ACC/AHA clinical practice guidelines.
        JAMA. 2009; 301: 831-841
        • Brown B.
        Delphi process: a methodology used for elicitation of opinions of experts.
        Rand Corporation, Santa Monica, CA, 1968: 3925-3940 (https://www.rand.org/pubs/papers/P3925.html)
        • Al-Hameed F.
        • Al-Dorzi H.M.
        • Shamy A.
        • et al.
        The Saudi clinical practice guideline for the diagnosis of the first deep venous thrombosis of the lower extremity.
        Ann Thorac Med. 2015; 10: 3-15
        • Al-Mandeel H.M.
        • Sagr E.
        • Sait K.
        • et al.
        Clinical practice guidelines on the screening and treatment of precancerous lesions for cervical cancer prevention in Saudi Arabia.
        Ann Saudi Med. 2016; 36: 313-320
        • Pai M.
        • Santesso N.
        • Yeung C.H.
        • Lane S.J.
        • Schunemann H.J.
        • Iorio A.
        Methodology for the development of the NHF-McMaster Guideline on care models for haemophilia management.
        Haemophilia. 2016; 22: 17-22
        • Coughlin S.S.
        Recall bias in epidemiologic studies.
        J Clin Epidemiol. 1990; 43: 87-91
        • Mustafa R.A.
        • Zimmerman D.
        • Rioux J.P.
        • Lindsay R.
        • Pierratos A.
        • Nesrallah G.E.
        Vascular access for intensive maintenance hemodialysis: a systematic review for a Canadian Society of Nephrology clinical practice guideline.
        Am J Kidney Dis. 2013; 62: 112-131
        • Nesrallah G.E.
        • Mustafa R.A.
        • MacRae J.
        • Pauly R.P.
        • Perkins D.N.
        • Gangji A.
        • et al.
        Canadian Society of Nephrology guidelines for the management of patients with ESRD treated with intensive hemodialysis.
        Am J Kidney Dis. 2013; 62: 187-198