Meta-analyses of diagnostic test accuracy could not be reproduced

  • Inge Stegeman
    Correspondence
    Corresponding author. Tel.: +31612435610.
    Affiliations
    Department of Otorhinolaryngology and Head & Neck Surgery, University Medical Center Utrecht, Utrecht, the Netherlands

    Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht, the Netherlands

    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Meibergdreef 9, Amsterdam, the Netherlands
    Search for articles by this author
  • Mariska M.G. Leeflang
    Affiliations
    Department of Clinical Epidemiology, Biostatistics and Bioinformatics, Amsterdam UMC, University of Amsterdam, Amsterdam Public Health, Meibergdreef 9, Amsterdam, the Netherlands
    Search for articles by this author

      Abstract

      Background and Objectives

      The aim of our study was to investigate the reproducibility of diagnostic accuracy meta-analyses, as reported in published systematic reviews.

      Study Design and Setting

      We selected all systematic reviews of diagnostic test accuracy containing a meta-analysis, published in January 2018 and retrieved in Medline through Ovid. All reviews reported a summary estimate of sensitivity and specificity. We requested the protocol from their authors and used the protocol and the information in the published review to reproduce the reported meta-analysis. Successful reproduction was defined as a result differing <1% point from the reported point estimates; or reported primary study results that were in line with those of the actual primary study results; or if the data from the primary studies could be extracted without checking the data in the review first.

      Results

      Of the 51 included reviews, 16 had a protocol registered in PROSPERO and five of those responded to our request for a protocol. Nineteen reviews (37%) provided the 2×2 tables that were included in the meta-analysis. In 14 of those, the outcome of the meta-analysis could be reproduced. Considering the correctness of the numbers from the primary articles and the complete reporting of the search strategy, only one meta-analysis was fully replicable.

      Conclusion

      Published meta-analyses of diagnostic test accuracy were poorly replicable. This was partly because of lack of information about the methods and data used, and partly because of mistakes in the data extraction or data reporting.

      Keywords

      To read this article in full you will need to make a payment

      References

        • Ioannidis J.P.
        Why most published research findings are false.
        PLoS Med. 2005; 2: 0696-0701
        • Ebrahim S.
        • Sohani Z.N.
        • Montoya L.
        • Agarwal A.
        • Thorlund K.
        • Mills E.J.
        • et al.
        Reanalyses of randomized clinical trial data.
        JAMA. 2014; 312: 1024
        • Besnik D.
        • Shamoo A.
        Reproducibility and research integrity.
        Acc Res. 2017; 24: 116-123
        • McMillan D.
        Replication studies.
        Cogent Economics and Finance; Royal Netherlands Academy of Arts and Sciences (KNAW), Amsterdam, Netherlands2017
        • Taichman D.B.
        • Sahni P.
        • Pinborg A.
        • Peiperl L.
        • Laine C.
        • James A.
        • et al.
        Data sharing statements for clinical trials — a requirement of the international committee of medical journal editors.
        N Engl J Med. 2017; 376: 2277-2279
        • Iqbal S.A.
        • Wallach J.D.
        • Khoury M.J.
        • Schully S.D.
        • Ioannidis J.P.
        Reproducible research Practices and transparency across the biomedical literature.
        PLoS Biol. 2016; 14: 1-13
        • Siontis K.C.
        • Hernandez-Boussard T.
        • Ioannidis J.P.A.
        Overlapping meta-analyses on the same topic: survey of published studies.
        BMJ. 2013; 347: 1-11
        • Pieper D.
        • Antoine S.
        • Mathes T.
        • Neugebauer E.A.M.
        • Eikermann M.
        Systematic review finds overlapping reviews were not mentioned in every other overview.
        J Clin Epidemiol. 2014; 67: 368-375
        • Leeflang M.M.G.
        • Deeks J.J.
        • Gatsonis C.A.
        • Bossuyt P.M.M.
        Systematic reviews of diagnostic test accuracy.
        Ann Intern Med. 2008; 149: 889-897
        • Page M.J.
        • Shamseer L.
        • Altman D.G.
        • Tetzlaff J.
        • Sampson M.
        • Tricco A.C.
        • et al.
        Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study.
        PLoS Med. 2016; 13: 1-30
        • StataCorp
        Stata Statistical Software: Release 15.
        StataCorp LLC, College Station, TX2017
        • Zamora J.
        • Abraira V.
        • Muriel A.
        • Khan K.
        • Coomarasamy A.
        Meta-DiSc: a software for meta-analysis of test accuracy data.
        BMC Med Res Methodol. 2006; 6: 1-12
        • Douglas B.
        • Maechler M.
        • Ben Bolker S.W.
        Fitting linear mixed-effects models using lme4.
        J Stat Softw. 2015; 67: 1-489
        • Schwarzer G.
        meta: an R package for meta-analysis.
        R News. 2007; 7: 40-45
        • Wallace B.C.
        • Dahabreh I.J.
        • Trikalinos T.A.
        • Lau J.
        • Paul T.
        • Schmid C.H.
        Closing the Gap between methodologists and end-users: R as a computational back-end.
        J Stat Softw. 2012; 49: 1-5
        • McInnes M.D.F.
        • Moher D.
        • Thombs B.D.
        • McGrath T.A.
        • Bossuyt P.M.
        • Clifford T.
        • et al.
        Preferred reporting items for a systematic review and meta-analysis of diagnostic test accuracy studies the PRISMA-DTA statement.
        JAMA. 2018; 319: 388-396