Meta-analyses of diagnostic test accuracy could not be reproduced

  • Inge Stegeman
    Corresponding author. Tel.: +31612435610.
    Department of Otorhinolaryngology and Head & Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands

    Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht, the Netherlands

    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Meibergdreef 9, Amsterdam, the Netherlands
    Search for articles by this author
  • Mariska M.G. Leeflang
    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Meibergdreef 9, Amsterdam, the Netherlands
    Search for articles by this author


      Background and Objectives

      The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews.

      Study Design and Setting

      We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through Ovid. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. Successful reproduction was defined as a result differing <1% point from the reported point estimates; or reported primary study results that were in line with those of the actual primary study results; or if the data from the primary studies could be extracted without checking the data in the review first.


      Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2×2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced. Considering the correctness of the numbers from the primary articles and the complete reporting of the search strategy, only one meta-analysis was fully replicable.


      Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used, and partly because of mistakes in the data extraction or data reporting.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Ioannidis J.P.
        Why most published research findings are false.
        PLoS Med. 2005; 2: 0696-0701
        • Ebrahim S.
        • Sohani Z.N.
        • Montoya L.
        • Agarwal A.
        • Thorlund K.
        • Mills E.J.
        • et al.
        Reanalyses of randomized clinical trial data.
        JAMA. 2014; 312: 1024
        • Besnik D.
        • Shamoo A.
        Reproducibility and research integrity.
        Acc Res. 2017; 24: 116-123
        • McMillan D.
        Replication studies.
        Cogent Economics and Finance; Royal Netherlands Academy of Arts and Sciences (KNAW), Amsterdam, Netherlands2017
        • Taichman D.B.
        • Sahni P.
        • Pinborg A.
        • Peiperl L.
        • Laine C.
        • James A.
        • et al.
        Data sharing statements for clinical trials — a requirement of the international committee of medical journal editors.
        N Engl J Med. 2017; 376: 2277-2279
        • Iqbal S.A.
        • Wallach J.D.
        • Khoury M.J.
        • Schully S.D.
        • Ioannidis J.P.
        Reproducible research Practices and transparency across the biomedical literature.
        PLoS Biol. 2016; 14: 1-13
        • Siontis K.C.
        • Hernandez-Boussard T.
        • Ioannidis J.P.A.
        Overlapping meta-analyses on the same topic: survey of published studies.
        BMJ. 2013; 347: 1-11
        • Pieper D.
        • Antoine S.
        • Mathes T.
        • Neugebauer E.A.M.
        • Eikermann M.
        Systematic review finds overlapping reviews were not mentioned in every other overview.
        J Clin Epidemiol. 2014; 67: 368-375
        • Leeflang M.M.G.
        • Deeks J.J.
        • Gatsonis C.A.
        • Bossuyt P.M.M.
        Systematic reviews of diagnostic test accuracy.
        Ann Intern Med. 2008; 149: 889-897
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.
        PLoS Med. 2016; 13: 1-30
        • StataCorp
        Stata Statistical Software: Release 15.
        StataCorp LLC, College Station, TX2017
        • Zamora J.
        • Abraira V.
        • Muriel A.
        • Khan K.
        • Coomarasamy A.
        Meta-DiSc: a software for meta-analysis of test accuracy data.
        BMC Med Res Methodol. 2006; 6: 1-12
        • Douglas B.
        • Maechler M.
        • Ben Bolker S.W.
        Fitting linear mixed-effects models using lme4.
        J Stat Softw. 2015; 67: 1-489
        • Schwarzer G.
        meta: an R package for meta-analysis.
        R News. 2007; 7: 40-45
        • Wallace B.C.
        • Dahabreh I.J.
        • Trikalinos T.A.
        • Lau J.
        • Paul T.
        • Schmid C.H.
        Closing the Gap between methodologists and end-users: R as a computational back-end.
        J Stat Softw. 2012; 49: 1-5
        • McInnes M.D.F.
        • Moher D.
        • Thombs B.D.
        • McGrath T.A.
        • Bossuyt P.M.
        • Clifford T.
        • et al.
        Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies the PRISMA-DTA statement.
        JAMA. 2018; 319: 388-396