Advertisement

Publication bias is underreported in systematic reviews published in high-impact-factor journals: metaepidemiologic study

  • Akira Onishi
    Correspondence
    Corresponding author. Tel.: +81-75-753-9491; fax: +81-75-753-4641.
    Affiliations
    Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501 Japan
    Search for articles by this author
  • Toshi A. Furukawa
    Affiliations
    Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501 Japan
    Search for articles by this author
Published:September 04, 2014DOI:https://doi.org/10.1016/j.jclinepi.2014.07.002

      Abstract

      Objectives

      To examine how often a significant publication bias (PB) existed when the assessment of PB was not reported in systematic reviews.

      Study Design and Setting

      All systematic reviews with meta-analyses of interventions and risk/prognostic factors published in the general medical journals with the top 10 impact factors in 2011 and 2012 were included. The results regarding PB were extracted. When the assessment of PB was not reported, we examined the presence of PB using the Egger test and contour-enhanced funnel plot and the impact of unreported PB by regression-based method.

      Results

      Among all the identified 116 reviews, the assessment of PB was not reported in 36 reviews (31.0%), particularly in reviews without a comprehensive literature search. Of these 36 reviews, seven (19.4%) were found to have a significant PB. The original pooled results may have been overestimated by a median of 50.9% if corrected for PB. Among the 28 reviews with PB including both reviews that did or did not report the assessment of PB, seven reviews (25.0%) did not report the presence of PB.

      Conclusion

      Significant PB was underreported in systematic reviews published in high-impact-factor journals (eg, 19.4% of those that did not report assessment of PB had significant PB). Readers of systematic reviews should not assume that PB does not exist when not reported whereas researchers should report the results of assessments for PB.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • Sterne J.
        • Egger M.
        • Smith G.D.
        Investigating and dealing with publication and other biases.
        in: Egger M. Smith G.D. Altman D.G. Systematic reviews in health care: meta-analysis in context. 2nd ed. BMJ Publishing Group, London, United Kingdom2001: 189-208
        • Song F.
        • Parekh S.
        • Hooper L.
        • Loke Y.K.
        • Ryder J.
        • Sutton A.J.
        • et al.
        Dissemination and publication of research findings: an updated review of related biases.
        Health Technol Assess. 2010; 14 (ix–xi, 1–193): iii
        • Sena E.S.
        • van der Worp H.B.
        • Bath P.M.
        • Howells D.W.
        • Macleod M.R.
        Publication bias in reports of animal stroke studies leads to major overstatement of efficacy.
        PLoS Biol. 2010; 8: e1000344
        • Moher D.
        • Tetzlaff J.
        • Tricco A.C.
        • Sampson M.
        • Altman D.G.
        Epidemiology and reporting characteristics of systematic reviews.
        PLoS Med. 2007; 4: e78
        • Moher D.
        • Cook D.J.
        • Eastwood S.
        • Olkin I.
        • Rennie D.
        • Stroup D.F.
        Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of Reporting of Meta-analyses.
        Lancet. 1999; 354: 1896-1900
        • Stroup D.F.
        • Berlin J.A.
        • Morton S.C.
        • Olkin I.
        • Williamson G.D.
        • Rennie D.
        • et al.
        Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group.
        JAMA. 2000; 283: 2008-2012
        • Moher D.
        • Liberati A.
        • Tetzlaff J.
        • Altman D.G.
        Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement.
        BMJ. 2009; 339: b2535
        • Lee K.P.
        • Schotland M.
        • Bacchetti P.
        • Bero L.A.
        Association of journal quality indicators with methodological quality of clinical research articles.
        JAMA. 2002; 287: 2805-2808
        • Korevaar D.A.
        • Hooft L.
        • ter Riet G.
        Systematic reviews and meta-analyses of preclinical studies: publication bias in laboratory animal experiments.
        Lab Anim. 2011; 45: 225-230
        • Yank V.
        • Rennie D.
        • Bero L.A.
        Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study.
        BMJ. 2007; 335: 1202-1205
        • Sterne J.
        • Egger M.
        • Moher D.
        Addressing reporting biases.
        in: Higgins J. Green S. Cochrane handbook for systematic reviews of intervention. Version 5.1.0 (updated March 2011). New York: The Cochrane Collaboration, 2011 ([Chapter 10]. Available at) (Accessed May 7, 2013)
      1. Web of knowledge. Available at http://www.thompson.isi.com. Accessed May 7, 2013.

        • Souza J.P.
        • Pileggi C.
        • Cecatti J.G.
        Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey.
        Reprod Health. 2007; 4: 3
        • Egger M.
        • Davey Smith G.
        • Schneider M.
        • Minder C.
        Bias in meta-analysis detected by a simple, graphical test.
        BMJ. 1997; 315: 629-634
        • Peters J.L.
        • Sutton A.J.
        • Jones D.R.
        • Abrams K.R.
        • Rushton L.
        Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry.
        J Clin Epidemiol. 2008; 61: 991-996
        • Mavridis D.
        • Salanti G.
        Exploring and accounting for publication bias in mental health: a brief overview of methods.
        Evid Based Ment Health. 2014; 17: 11-15
        • Shea B.J.
        • Grimshaw J.M.
        • Wells G.A.
        • Boers M.
        • Andersson N.
        • Hamel C.
        • et al.
        Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews.
        BMC Med Res Methodol. 2007; 7: 10
        • Shea B.J.
        • Hamel C.
        • Wells G.A.
        • Bouter L.M.
        • Kristjansson E.
        • Grimshaw J.
        • et al.
        AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews.
        J Clin Epidemiol. 2009; 62: 1013-1020
        • Pimouguet C.
        • Le Goff M.
        • Thiebaut R.
        • Dartigues J.F.
        • Helmer C.
        Effectiveness of disease-management programs for improving diabetes care: a meta-analysis.
        CMAJ. 2011; 183: E115-E127
        • Sterne J.A.
        • Sutton A.J.
        • Ioannidis J.P.
        • Terrin N.
        • Jones D.R.
        • Lau J.
        • et al.
        Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials.
        BMJ. 2011; 343: d4002
        • Vavken P.
        • Dorotka R.
        The prevalence and effect of publication bias in orthopaedic meta-analyses.
        J Orthop Sci. 2011; 16: 238-244
        • Turner E.H.
        • Knoepflmacher D.
        • Shapley L.
        Publication bias in antipsychotic trials: an analysis of efficacy comparing the published literature to the US Food and Drug Administration database.
        PLoS Med. 2012; 9: e1001189
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • Dwan K.
        • Altman D.G.
        • Arnaiz J.A.
        • Bloom J.
        • Chan A.W.
        • Cronin E.
        • et al.
        Systematic review of the empirical evidence of study publication bias and outcome reporting bias.
        PLoS One. 2008; 3: e3081