Review Article| Volume 64, ISSUE 4, P349-357, April 2011

Download started.


Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement

Published:September 01, 2010DOI:



      To evaluate the measures taken to deal with publication bias across different categories of systematic reviews published in 2006 and to compare these with reviews published in 1996.

      Study Design and Setting

      PubMed was searched for systematic reviews published in 2006; 100 treatment effect, 50 diagnostic accuracy, 100 risk factor, and 50 gene–disease association reviews were randomly selected.


      The use of MEDLINE increased from 74% to 95%; checking references increased from 42% to 73%; use of Cochrane Library increased from 5% to 58%; and use of CINAHL increased from 8% in 1996 to 24% in treatment reviews, 20% in diagnostic reviews, 18% in risk factor reviews, and 0% in genetic reviews published in 2006. A 20% increase was observed for explicit searching of non–English-language studies in all reviews published in 2006. Efforts to search for unpublished studies increased to 61% from 35% in treatment reviews published in 1996. Twenty-six percent of the reviews used funnel plots or related methods to test for publication bias compared with less than 6% in earlier reviews.


      Recent reviews show a significant improvement in the measures taken to prevent publication bias. However, few methods exist to deal with publication bias in the nonquantitative findings of systematic reviews.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Dickersin K.
        The existence of publication bias and risk factors for its occurrence.
        JAMA. 1990; 263: 1385-1389
        • Song F.
        • Parekh S.
        • Hooper L.
        • Loke Y.
        • Ryder J.J.
        • Sutton A.J.
        • et al.
        Dissemination and publication of research findings: an updated review of related biases.
        Health Technol Assess. 2010; 14: 1-219
        • Song F.
        • Eastwood A.J.
        • Gilbody S.
        • Duley L.
        • Sutton A.J.
        Publication and related biases.
        Health Technol Assess. 2000; 4: 1-115
        • Landis J.R.
        • Koch G.G.
        The measurement of observer agreement for categorical data.
        Biometrics. 1977; 33: 159-174
        • Lowry R.
        (Available at) (Accessed Dec 4, 2009)
        • McDonald S.
        • Taylor L.
        • Adams C.
        Searching the right database. A comparison of four databases for psychiatry journals.
        Health Libr Rev. 1999; 16: 151-156
        • Lefebvre C.
        • Manheimer E.
        • Glanville J.
        Searching for studies.
        in: Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Interventions. Version 5.0.2 (updated September 2009). The Cochrane Collaboration, 2009 (Available at) (Accessed Dec 4, 2009)
        • Gulmezoglu A.M.
        • Pang T.
        • Horton R.
        • Dickersin K.
        WHO facilitates international collaboration in setting standards for clinical trial registration.
        Lancet. 2005; 365: 1829-1831
        • DeAngelis C.D.
        • Drazen J.M.
        • Frizelle F.A.
        • Haug C.
        • Hoey J.
        • Horton R.
        • et al.
        Clinical trial registration: a statement from the International Committee of Medical Journal Editors.
        JAMA. 2004; 292: 1363-1364
        • Abbasi K.
        Compulsory registration of clinical trials.
        BMJ. 2004; 329: 637-638
        • Laine C.
        • Horton R.
        • DeAngelis C.D.
        • Drazen J.M.
        • Frizelle F.A.
        • Godlee F.
        • et al.
        Clinical trial registration: looking back and moving ahead.
        Lancet. 2007; 369: 1909-1911
        • Whittington C.J.
        • Kendall T.
        • Fonagy P.
        • Cottrell D.
        • Cotgrove A.
        • Boddington E.
        Selective serotonin reuptake inhibitors in childhood depression: systematic review of published versus unpublished data.
        Lancet. 2004; 363: 1341-1345
        • Batt K.
        • Rushby J.A.F.
        • Riquelme M.C.
        The costs, effects and cost-effectiveness of strategies to increase coverage of routine immunizations in low- and middle-income countries: systematic review of the grey literature.
        Bull World Health Organ. 2004; 82: 689-696
        • Turner E.H.
        • Matthews A.M.
        • Linardatos E.
        • Tell R.A.
        • Rosenthal R.
        Selective publication of antidepressant trials and its influence on apparent efficacy.
        N Engl J Med. 2008; 358: 252-260
        • van Driel M.L.
        • De Sutter A.
        • De Maeseneer J.
        • Christiaens T.
        Searching for unpublished trials in Cochrane reviews may not be worth the effort.
        J Clin Epidemiol. 2009; 62: 838-844.e3
        • Davey-Smith G.
        • Egger M.
        Meta-analysis: unresolved issues and future developments.
        BMJ. 1998; 316: 221-225
        • Egger M.
        • Juni P.
        • Bartlett C.
        • Holenstein F.
        • Sterne J.
        How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study.
        Health Technol Assess. 2003; 7: 1-76
        • Moher D.
        • Pham B.
        • Lawson M.L.
        • Klassen T.P.
        The inclusion of reports of randomised trials published in languages other than English in systematic reviews.
        Health Technol Assess. 2003; 7: 1-90
        • Heres S.
        • Wagenpfeil S.
        • Hamann J.
        • Kissling W.
        • Leucht S.
        Language bias in neuroscience—is the Tower of Babel located in Germany?.
        Eur Psychiatry. 2004; 19: 230-232
        • Brazzelli M.
        • Lewis S.C.
        • Deeks J.J.
        • Sandercock P.A.G.
        No evidence of bias in the process of publication of diagnostic accuracy studies in stroke submitted as abstracts.
        J Clin Epidemiol. 2009; 62: 425-430
      1. Little J, Higgins JPT, editors. The HuGENet™ HuGE review handbook, version 1.0. Available at Accessed Dec 4, 2009.

        • Pan Z.
        • Trikalinos T.A.
        • Kavvoura F.K.
        • Lau J.
        • Ioannidis J.P.A.
        Local literature bias in genetic epidemiology: an empirical evaluation of the Chinese literature.
        PLoS Med. 2005; 2 ([see comment]): e334
        • Lau J.
        • Ioannidis J.P.
        • Terrin N.
        • Schmid C.H.
        • Olkin I.
        The case of the misleading funnel plot.
        BMJ. 2006; 333: 597-600
        • Ioannidis J.P.A.
        • Trikalinos T.A.
        The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey.
        CMAJ. 2007; 176: 1091-1096
        • Sterne J.A.C.
        • Egger M.
        • Moher D.
        Addressing reporting biases.
        in: Higgins J.P.T. Green S. Cochrane Handbook for Systematic Reviews of Intervention. Version 5.0.1 (updated September 2008). The Cochrane Collaboration, 2008 (Available at) (Accessed Dec 4, 2009)