Preferred reporting items for overviews of systematic reviews including harms checklist: a pilot tool to be used for balanced reporting of benefits and harms

  • Konstantinos I. Bougioukas
    Department of Hygiene and Epidemiology, Medical School, Aristotle University of Thessaloniki, University Campus, 54124 Thessaloniki, Greece
    Search for articles by this author
  • Aris Liakos
    Second Medical Department, Clinical Research and Evidence-Based Medicine Unit, Aristotle University of Thessaloniki, Konstantinoupoleos 49, 54642 Thessaloniki, Greece
    Search for articles by this author
  • Apostolos Tsapas
    Second Medical Department, Clinical Research and Evidence-Based Medicine Unit, Aristotle University of Thessaloniki, Konstantinoupoleos 49, 54642 Thessaloniki, Greece

    Harris Manchester College, University of Oxford, Mansfield Rd, Oxford, OX1 3TD, UK
    Search for articles by this author
  • Evangelia Ntzani
    Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, University Campus, Stavros Niarchos Av., Ioannina, Greece

    Center for Evidence Synthesis in Health, Department of Health Services, Policy and Practice, School of Public Health, Brown University, 121 South Main Street, Providence, RI 02903, USA
    Search for articles by this author
  • Anna-Bettina Haidich
    Corresponding author. Tel.: +30-2310999143; fax: +302310999701.
    Department of Hygiene and Epidemiology, Medical School, Aristotle University of Thessaloniki, University Campus, 54124 Thessaloniki, Greece
    Search for articles by this author



      An overview of systematic reviews (OoSRs) is a study designed to synthesize multiple evidence from existing systematic reviews on a specific domain. The aim of this paper was to offer a pilot version checklist with Preferred Reporting Items for OoSRs (PRIO-harms) to promote a more balanced reporting of benefits and harms in OoSRs of health care interventions.

      Study Design and Setting

      The included items were developed by combining key features from health care OoSRs designs with recommendations from statements of other relevant checklists and pertinent methodological review articles. Two raters independently used the PRIO-harms checklist to assess a sample of 20 OoSRs.


      The PRIO-harms tool consists of a 27-item (56 (sub-)items in total) checklist and is accompanied by a five-stage process flow diagram (identification, screening, eligibility, inclusion, and separation of relevant studies). The mean interrater reliability (Gwet's AC1 statistic) between reviewers was 0.90 (95% confidence interval: 0.88, 0.92) indicating a very good agreement.


      The PRIO-harms tool can be used in every OoSRs that addresses health care interventions. This instrument will assist overview authors to improve completeness and transparency of research reporting with emphasis on harms. However, it might benefit from critical review and further validation from experts and research teams that produce OoSRs.


      To read this article in full you will need to make a payment


        • Becker A.L.
        • Oxman D.A.
        Chapter 22: overviews of reviews.
        in: Higgins J. Green S. Cochrane Handb. Syst. Rev. Interv. Version 5.1.0 (updated March 2011). The Cochrane Collaboration, 2011 (Available at Accessed November 6, 2017)
        • Lavis J.N.
        How can we support the use of systematic reviews in policymaking?.
        PLoS Med. 2009; 6: e1000141
        • Thomson D.
        • Russell K.
        • Becker L.
        • Klassen T.
        • Hartling L.
        The evolution of a new publication type: steps and challenges of producing overviews of reviews.
        Res Synth Methods. 2010; 1: 198-211
        • Murad M.H.
        • Asi N.
        • Alsawas M.
        • Alahdab F.
        New evidence pyramid.
        Evid Based Med. 2016; 21: 125-127
        • Lemeshow A.R.
        • Blum R.E.
        • Berlin J.A.
        • Stoto M.A.
        • Colditz G.A.
        Searching one or two databases was insufficient for meta-analysis of observational studies.
        J Clin Epidemiol. 2005; 58: 867-873
        • Golder S.
        • Loke Y.
        • McIntosh H.M.
        Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects.
        J Clin Epidemiol. 2008; 61: 440-448
        • Pieper D.
        • Buechter R.
        • Jerinic P.
        • Eikermann M.
        Overviews of reviews often have limited rigor: a systematic review.
        J Clin Epidemiol. 2012; 65: 1267-1273
        • Hartling L.
        • Vandermeer B.
        • Fernandes R.M.
        Systematic reviews, overviews of reviews and comparative effectiveness reviews: a discussion of approaches to knowledge synthesis.
        Evid Based Child Health. 2014; 9: 486-494
        • Pieper D.
        • Antoine S.L.
        • Mathes T.
        • Neugebauer E.A.M.
        • Eikermann M.
        Systematic review finds overlapping reviews were not mentioned in every other overview.
        J Clin Epidemiol. 2014; 67: 368-375
        • Hartling L.
        • Chisholm A.
        • Thomson D.
        • Dryden D.M.
        A descriptive analysis of overviews of reviews published between 2000 and 2011.
        PLoS One. 2012; 7: e49667
        • Pieper D.
        • Antoine S.L.
        • Morfeld J.C.
        • Mathes T.
        • Eikermann M.
        Methodological approaches in conducting overviews: current state in HTA agencies.
        Res Synth Methods. 2014; 5: 187-199
        • Pieper D.
        • Antoine S.L.
        • Neugebauer E.A.M.
        • Eikermann M.
        Up-to-dateness of reviews is often neglected in overviews: a systematic review.
        J Clin Epidemiol. 2014; 67: 1302-1308
        • Haidich A.B.
        • Birtsou C.
        • Dardavessis T.
        • Tirodimos I.
        • Arvanitidou M.
        The quality of safety reporting in trials is still suboptimal: survey of major general medical journals.
        J Clin Epidemiol. 2011; 64: 124-135
        • Saini P.
        • Loke Y.K.
        • Gamble C.
        • Altman D.G.
        • Williamson P.R.
        • Kirkham J.J.
        Selective reporting bias of harm outcomes within studies: findings from a cohort of systematic reviews.
        BMJ. 2014; 349: g6501
        • Mahady S.E.
        • Schlub T.
        • Bero L.
        • Moher D.
        • Tovey D.
        • George J.
        • et al.
        Side effects are incompletely reported among systematic reviews in gastroenterology.
        J Clin Epidemiol. 2015; 68: 144-153
        • Zorzela L.
        • Golder S.
        • Liu Y.
        • Pilkington K.
        • Hartling L.
        • Joffe A.
        • et al.
        Quality of reporting in systematic reviews of adverse events: systematic review.
        BMJ. 2014; 348: f7668
        • Wallace B.C.
        • Dahabreh I.J.
        • Schmid C.H.
        • Lau J.
        • Trikalinos T.A.
        Modernizing the systematic review process to inform comparative effectiveness: tools and methods.
        J Comp Eff Res. 2013; 2: 273-282
        • Pollock M.
        • Fernandes R.M.
        • Becker L.A.
        • Featherstone R.
        • Hartling L.
        What guidance is available for researchers conducting overviews of reviews of healthcare interventions? A scoping review and qualitative metasummary.
        Syst Rev. 2016; 5: 190
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        • Group P.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339: b2535
        • Zorzela L.
        • Loke Y.K.
        • Ioannidis J.P.
        • Golder S.
        • Santaguida P.
        • Altman D.G.
        • et al.
        PRISMA harms checklist: improving harms reporting in systematic reviews.
        BMJ. 2016; 352: i157
        • Moher D.
        • Shamseer L.
        • Clarke M.
        • Ghersi D.
        • Liberati A.
        • Petticrew M.
        • et al.
        Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement.
        Syst Rev. 2015; 4: 1
        • Li L.
        • Tian J.
        • Tian H.
        • Sun R.
        • Liu Y.
        • Yang K.
        Quality and transparency of overviews of systematic reviews.
        J Evid Based Med. 2012; 5: 166-173
        • Onishi A.F.T.
        State of the art reporting.
        in: Biondi-Zoccai G. Umbrella Rev. Evid. Synth. with Overviews Rev. Meta-Epidemiologic Stud. Springer, London, UK2016: 189-202
        • Aromataris E.
        • Fernandez R.
        • Godfrey C.M.
        • Holly C.
        • Khalil H.
        • Tungpunkom P.
        Summarizing systematic reviews: methodological development, conduct and reporting of an umbrella review approach.
        Int J Evid Based Healthc. 2015; 13: 132-140
        • Caird J.
        • Sutcliffe K.
        • Kwan I.
        • Dickson K.T.J.
        Mediating policy-relevant evidence at speed: are systematic reviews of systematic reviews a useful approach?.
        Evid Policy. 2015; 11: 81-97
        • Smith V.
        • Devane D.
        • Begley C.M.
        • Clarke M.
        Methodology in conducting a systematic review of systematic reviews of healthcare interventions.
        BMC Med Res Methodol. 2011; 11: 15
        • Ballard M.
        • Montgomery P.
        Risk of bias in overviews of reviews: a scoping review of methodological guidance and four-item checklist.
        Res Synth Methods. 2017; 8: 92-108
        • Fix O.K.
        • Liou I.
        • Karvellas C.J.
        • Ganger D.R.
        • Forde K.A.
        • Subramanian R.M.
        • et al.
        Development and pilot of a checklist for management of Acute Liver Failure in the Intensive care Unit.
        PLoS One. 2016; 11: e0155500
        • Pollock M.
        • Sinha I.P.
        • Hartling L.
        • Rowe B.H.
        • Schreiber S.
        • Fernandes R.M.
        Inhaled short-acting bronchodilators for managing emergency childhood asthma: an overview of reviews.
        Allergy. 2017; 72: 183-200
        • Gwet K.L.
        Computing inter-rater reliability and its variance in the presence of high agreement.
        Br J Math Stat Psychol. 2008; 61: 29-48
        • Wongpakaran N.
        • Wongpakaran T.
        • Wedding D.
        • Gwet K.L.
        A comparison of Cohen's Kappa and Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples.
        BMC Med Res Methodol. 2013; 13: 61
        • El Emam K.
        Benchmarking Kappa: interrater agreement in software process assessments.
        Empir Softw Eng. 1999; 4: 113-133
        • Gwet K.L.
        Handbook of inter-rater reliability: the definitive guide to measuring the extent of agreement among raters.
        4th ed. Advanced Analytics, LLC, Gaithersburg2014
        • Martire RLo
        rel: Reliability coefficients. R package version 1.3.0.
        2017 (Available at Accessed November 6, 2017)
        • Büchter R.B.
        • Pieper D.
        Most overviews of Cochrane reviews neglected potential biases from dual authorship.
        J Clin Epidemiol. 2016; 77: 91-94
        • Shojania K.G.
        • Sampson M.
        • Ansari M.T.
        • Ji J.
        • Doucette S.
        • Moher D.
        Hοw quickly do systematic reviews go out of date? a survival analysis.
        Ann Intern Med. 2007; 147: 224-233
        • McNutt M.
        • Bradford M.
        • Drazen J.
        • Hanson R.B.
        • Howard B.
        • Jamieson K.H.
        • et al.
        Transparency in authors' contributions and responsibilities to promote integrity in scientific publication.
        bioRxiv. 2017; (Available at Accessed November 6, 2017)
        • Ioannidis J.P.A.
        • Evans S.J.W.
        • Gøtzsche P.C.
        • O'Neill R.T.
        • Altman D.G.
        • Schulz K.
        • et al.
        Better reporting of harms in randomized trials: an extension of the CONSORT statement.
        Ann Intern Med. 2004; 141: 781-788
        • Chou R.
        • Aronson N.
        • Atkins D.
        • Ismaila A.S.
        • Santaguida P.
        • Smith D.H.
        • et al.
        Assessing harms when comparing medical interventions.
        2008 (Rockville, MD. Available at Accessed November 6, 2017)
        • Sweeting M.J.
        • Sutton A.J.
        • Lambert P.C.
        What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data.
        Stat Med. 2004; 23: 1351-1375
        • Warren F.C.
        • Abrams K.R.
        • Golder S.
        • Sutton A.J.
        Systematic review of methods used in meta-analyses where a primary outcome is an adverse or unintended event.
        BMC Med Res Methodol. 2012; 12: 64
        • Cooper H.
        • Koenka A.C.
        The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship.
        Am Psychol. 2012; 67: 446-462
        • Siontis K.C.
        • Hernandez-Boussard T.
        • Ioannidis J.P.A.
        Overlapping meta-analyses on the same topic: survey of published studies.
        BMJ. 2013; 347: f4501
        • Jadad A.R.
        • Moore R.A.
        • Carroll D.
        • Jenkinson C.
        • Reynolds D.J.
        • Gavaghan D.J.
        • et al.
        Assessing the quality of reports of randomized clinical trials: is blinding necessary?.
        Control Clin Trials. 1996; 17: 1-12
        • Higgins J.P.T.
        • Altman D.G.
        • Gøtzsche P.C.
        • Jüni P.
        • Moher D.
        • Oxman A.D.
        • et al.
        The Cochrane Collaboration's tool for assessing risk of bias in randomised trials.
        BMJ. 2011; 343: d5928
        • Sterne J.A.
        • Hernan M.A.
        • Reeves B.C.
        • Savovic J.
        • Berkman N.D.
        • Viswanathan M.
        • et al.
        ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.
        BMJ. 2016; 355: i4919
      1. Wells G, Shea B, O'Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle-Ottawa scale (NOS) for assessing the quality of nonrandomized studies in meta-analysis. Ottawa, ON The Ottawa Health Research Institute. Available at Accessed July 15, 2017. [n.d]

        • Zhou X.H.
        • Perkins A.J.
        • Hui S.L.
        Comparisons of software packages for generalized linear multilevel models.
        Am Stat. 1999; 53: 282-290
        • Bergmann R.
        • Ludbrook J.
        • Spooren W.P.J.M.
        Different outcomes of the Wilcoxon-Mann-Whitney test from different statistics packages.
        Am Stat. 2000; 54: 72-77
        • McDonagh M.
        • Peterson K.
        • Raina P.
        • Chang S.
        • Shekelle P.
        Avoiding bias in selecting studies.
        2008 (Rockville, MD. Available at Accessed November 6, 2017)
        • Zintzaras E.
        Gamma-aminobutyric acid A receptor, alpha-2 (GABRA2) variants as individual markers for alcoholism: a meta-analysis.
        Psychiatr Genet. 2012; 22: 189-196
        • Niv Y.
        Estrogen receptor beta expression and colorectal cancer: a systematic review and meta-analysis.
        Eur J Gastroenterol Hepatol. 2015; 27: 1438-1442
        • Al-Hamoudi N.
        Is antimicrobial photodynamic therapy an effective treatment for chronic periodontitis in diabetes mellitus and cigarette smokers: a systematic review and meta-analysis.
        Photodiagnosis Photodyn Ther. 2017; 19: 375-382
        • Hamley S.
        The effect of replacing saturated fat with mostly n-6 polyunsaturated fat on coronary heart disease: a meta-analysis of randomised controlled trials.
        Nutr J. 2017; 16: 30
        • Puljak L.
        If there is only one author or only one database was searched, a study should not be called a systematic review.
        J Clin Epidemiol. 2017; ( [Epub ahead of print])
        • Dosenovic S.
        • Jelicic Kadic A.
        • Miljanovic M.
        • Biocic M.
        • Boric K.
        • Cavar M.
        • et al.
        Interventions for Neuropathic Pain: an overview of systematic reviews.
        Anesth Analg. 2017; 125: 643-652
        • Gajsak L.R.
        • Gelemanovic A.
        • Kuzman M.R.
        • Puljak L.
        Impact of stress response in development of first-episode psychosis in schizophrenia: an overview of systematic reviews.
        Psychiatr Danub. 2017; 29: 14-23
        • Pollock M.
        • Fernandes R.M.
        • Hartling L.
        Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions.
        BMC Med Res Methodol. 2017; 17: 48
        • Whiting P.
        • Savović J.
        • Higgins J.P.T.
        • Caldwell D.M.
        • Reeves B.C.
        • Shea B.
        • et al.
        ROBIS: a new tool to assess risk of bias in systematic reviews was developed.
        J Clin Epidemiol. 2016; 69: 225-234
        • Bühn S.
        • Mathes T.
        • Prengel P.
        • Wegewitz U.
        • Ostermann T.
        • Robens S.
        • et al.
        The risk of bias in systematic reviews (ROBIS) tool showed fair reliability and good construct validity.
        J Clin Epidemiol. 2017; ( [Epub ahead of print])
        • Ioannidis J.P.A.
        Adverse events in randomized trials: neglected, restricted, distorted, and silenced.
        Arch Intern Med. 2009; 169: 1737-1739
        • Pitrou I.
        • Boutron I.
        • Ahmad N.
        • Ravaud P.
        Reporting of safety results in published reports of randomized controlled trials.
        Arch Intern Med. 2009; 169: 1756-1761
        • Jonsson U.
        • Alaie I.
        • Parling T.
        • Arnberg F.K.
        Reporting of harms in randomized controlled trials of psychological interventions for mental and behavioral disorders: a review of current practice.
        Contemp Clin Trials. 2014; 38: 1-8
        • Hadi M.A.
        • McHugh G.A.
        • Conaghan P.G.
        Quality of reporting of harms in randomised controlled trials of pharmacological interventions for rheumatoid arthritis: a systematic review.
        Evid Based Med. 2017; 22: 170-177
        • Hodkinson A.
        • Kirkham J.J.
        • Tudur-Smith C.
        • Gamble C.
        Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension.
        BMJ Open. 2013; 3: e003436
        • Haidich A.B.
        • Pilalas D.
        • Contopoulos-Ioannidis D.G.
        • Ioannidis J.P.A.
        Most meta-analyses of drug interventions have narrow scopes and many focus on specific agents.
        J Clin Epidemiol. 2013; 66: 371-378
        • Lu G.
        • Ades A.E.
        Combination of direct and indirect evidence in mixed treatment comparisons.
        Stat Med. 2004; 23: 3105-3124
        • Lee A.W.
        Review of mixed treatment comparisons in published systematic reviews shows marked increase since 2009.
        J Clin Epidemiol. 2014; 67: 138-143
        • Ioannidis J.P.A.
        Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses.
        CMAJ. 2009; 181: 488-493
        • Cahill K.
        • Stevens S.
        • Perera R.
        • Lancaster T.
        Pharmacological interventions for smoking cessation: an overview and network meta-analysis.
        Cochrane Database Syst Rev. 2013; 5: CD009329
        • Xin Y.
        • Manson J.
        • Govan L.
        • Harbour R.
        • Bennison J.
        • Watson E.
        • et al.
        Pharmacological regimens for eradication of Helicobacter pylori: an overview of systematic reviews and network meta-analysis.
        BMC Gastroenterol. 2016; 16: 80
        • Hutton B.
        • Salanti G.
        • Caldwell D.M.
        • Chaimani A.
        • Schmid C.H.
        • Cameron C.
        • et al.
        The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations.
        Ann Intern Med. 2015; 162: 777-784