Advertisement

Publication bias in meta-analysis

its causes and consequences

      Abstract

      Publication bias is a widespread problem that may seriously distort attempts to estimate the effect under investigation. The literature is reviewed to determine features of the design and execution of both single studies and meta-analyses leading to publication bias, and the role the author, journal editor, and reviewer play in selecting studies for publication. Methods of detecting, correcting for, and preventing publication bias are reviewed. The design of the meta-analysis itself, and the studies included in it, are shown to be important among a number of sources of publication bias. Various factors influence an author's decision to submit results for publication. Journal editors and reviewers are crucial in deciding which studies to publish. Various methods proposed for detecting and correcting for publication bias, though useful, all have limitations. However, prevention of publication bias by registering every trial undertaken or publishing all studies is an ideal that is hard to achieve.

      Keywords

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Subscribe:

      Subscribe to Journal of Clinical Epidemiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

      1. Last J.M. A Dictionary of Epidemiology. Oxford University Press, Oxford1983 (p. 12.)
        • Rosenthal R.
        The “file drawer problem” and tolerance for null results.
        Psychol Bull. 1979; 86: 638-641
        • Smith M.B.
        Editorial.
        J Abnorm Social Psychol. 1956; 52: 1-4
        • Sterling T.
        Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa.
        Am Stat Assoc J. 1959; 54: 30-34
        • Smart R.G.
        The importance of negative results in psychological research.
        Can Psychol. 1964; 5: 225-232
        • Tweedie R.L.
        • Scott D.J.
        • Biggerstaff B.J.
        • Mengersen K.L.
        Bayesian meta-analysis, with application to studies of ETS and lung cancer.
        Lung Cancer. 1996; 14: S171-S194
        • Angell M.
        Negative studies.
        N Engl J Med. 1989; 321: 464-466
        • Newcombe R.G.
        Towards a reduction in publication bias.
        BMJ. 1987; 295: 656-659
        • Pocock S.J.
        • Hughes M.D.
        • Lee R.J.
        Statistical problems in the reporting of clinical trials.
        N Engl J Med. 1987; 317: 426-432
        • Green S.J.
        • Fleming T.R.
        • Emerson S.
        Effects on overviews of early stopping rules for clinical trials.
        Stat Med. 1987; 6: 361-369
        • Felson D.T.
        Bias in meta-analytic research.
        J Clin Epidemiol. 1992; 45: 885-892
        • Lee P.N.
        Problems in interpreting epidemiological data.
        in: Mohr U. Bates D.V. Dungworth D.L. Lee P.N. McClellan R.O. Roe F.J.C. Assessment of Inhalation Hazards. Springer-Verlag, Berlin1989: 49-50
        • Begg C.B.
        • Berlin J.A.
        Publication bias.
        J R Stat Soc A. 1988; 151: 419-463
        • Givens G.H.
        • Smith D.D.
        • Tweedie R.L.
        Publication bias in meta-analysis.
        Stat Sci. 1997; 12: 221-250
        • Engler R.I.
        • Covell J.W.
        • Friedman P.J.
        • Kitcher P.S.
        • Peters R.M.
        Misrepresentation and responsibility in medical research.
        N Engl J Med. 1983; 317: 1383-1389
        • Rosenthal R.
        • Fode K.L.
        Psychology of the scientist.
        Psychol Rep. 1963; 12: 491-511
        • Rosenthal R.
        Experimenter outcome-orientation and the results of the psychological experiment.
        Psychol Bull. 1964; 61: 405-412
        • Chalmers T.C.
        • Berrier J.
        • Sacks H.S.
        • Levin H.
        • Reitman D.
        • Nagalingam R.
        Meta-analysis of clinical trials as a scientific discipline. I.
        Stat Med. 1987; 6: 315-328
        • Dickersin K.
        The existence of publication bias and risk factors for its occurrence.
        JAMA. 1990; 263: 1385-1389
        • Kleijnen J.
        • Knipschild P.
        Review articles and publication bias.
        Arzneimittelforschung. 1992; 42: 587-591
        • Minerva
        Re publication bias.
        BMJ. 1992; 304: 264
        • Light R.J.
        Accumulating evidence from independent studies.
        Stat Med. 1987; 6: 221-231
        • Begg C.B.
        A measure to aid in the interpretation of published clinical trials.
        Stat Med. 1985; 4: 1-9
        • Bulpitt C.J.
        Meta-analysis.
        Lancet. 1988; ii: 93-94
        • Koren G.
        • Shear H.
        • Graham K.
        • Einarson T.
        Bias against the null hypothesis.
        Lancet. 1989; ii: 1440-1442
        • Chalmers T.C.
        • Frank C.S.
        • Reitman D.
        Minimizing the three stages of publication bias.
        JAMA. 1990; 263: 1392-1395
        • Charlton B.
        Think negative.
        New Scientist. 1987; 29Oct: 72
        • Mahoney M.J.
        Publication prejudices.
        Cog Ther Res. 1977; 1: 161-175
        • LeVois M.E.
        • Layard M.W.
        Publication bias in the environmental tobacco smoke/coronary heart disease epidemiologic literature.
        Regul Toxicol Pharmacol. 1995; 21: 184-191
        • Bero L.A.
        • Galbraith A.
        • Rennie D.
        Sponsored symposia on environmental tobacco smoke.
        JAMA. 1994; 271: 612-617
        • L'Abbé K.A.
        • Detsky A.S.
        • O'Rourke K.
        Meta-analysis in clinical research.
        Ann Intern Med. 1987; 107: 224-233
        • Mosteller F.
        • Colditz G.A.
        Understanding research synthesis (meta-analysis).
        Annu Rev Public Health. 1996; 17: 1-23
        • Easterbrook P.J.
        • Berlin J.A.
        • Gopalan R.
        • Matthews D.R.
        Publication bias in clinical research.
        Lancet. 1991; 337: 867-872
        • Blair A.
        • Burg J.
        • Foran J.
        • Gibb H.
        • Greenland S.
        • Morris R.
        • Raabe G.
        • Savitz D.
        • Teta J.
        • Wartenberg D.
        • Wong O.
        • Zimmerman R.
        Guidelines for application of meta-analysis in environmental epidemiology.
        Regul Toxicol Pharmacol. 1995; 22: 189-197
        • Greenland S.
        Quantitative methods in the review of epidemiologic literature.
        Epidemiol Rev. 1987; 9: 1-30
        • Sacks H.S.
        • Berrier J.
        • Reitman D.
        • Ancona-Berk V.A.
        • Chalmers T.C.
        Meta-analyses of randomized controlled trials.
        N Engl J Med. 1987; 316: 450-455
      2. Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? JAMA 1993;269:2749–53.

        • Kilpatrick S.J.
        The epidemiology of environmental tobacco smoke (ETS) and the weight of evidence argument.
        Int Surg. 1992; 77: 131-133
        • Pignon J.-P.
        • Arriagada R.
        Meta-analyses of randomized clinical trials.
        Lung Cancer. 1994; 10: S135-S141
        • Wald N.J.
        • Nanchahal K.
        • Thompson S.G.
        • Cuckle H.S.
        Does breathing other people's tobacco smoke cause lung cancer?.
        BMJ. 1986; 293: 1217-1222
        • Jones D.R.
        Meta-analysis of observational epidemiological studies.
        J R Soc Med. 1992; 85: 165-168
        • DerSimonian R.
        • Laird N.
        Meta-analysis in clinical trials.
        Controlled Clin Trials. 1986; 7: 177-188
        • Moher D.
        • Fortin P.
        • Jadad A.R.
        • Jüni P.
        • Klassen T.
        • Le Lorier J.
        • Liberati A.
        • Linde K.
        • Penna A.
        Completeness of reporting trials published in languages other than English.
        Lancet. 1996; 347: 363-366
        • Egger M.
        • Davey Smith G.
        • Schneider M.
        • Minder C.
        Bias in meta-analysis detected by a simple, graphical test.
        BMJ. 1997; 315: 629-634
        • Grégoire G.
        • Derderian F.
        • Le Lorier J.
        Selecting the language of the publications included in a meta-analysis.
        J Clin Epidemiol. 1995; 48: 159-163
        • Vickers A.
        • Goyal N.
        • Harland R.
        • Rees R.
        Do certain countries produce only positive results? A systematic review of controlled trials.
        Controlled Clin Trials. 1998; 19: 159-166
        • Gøtzsche P.C.
        Reference bias in reports of drug trials.
        BMJ. 1987; 295: 654-656
        • Tramèr M.R.
        • Reynolds D.J.M.
        • Moore R.A.
        • McQuay H.J.
        Impact of covert duplicate publication on meta-analysis.
        BMJ. 1997; 315: 635-640
        • Morris R.D.
        Meta-analysis in cancer epidemiology.
        Environ Health Perspect. 1994; 102: 61-66
        • Cappelleri J.C.
        • Ioannidis J.P.A.
        • Schmid C.H.
        • de Ferranti S.D.
        • Aubert M.
        • Chalmers T.C.
        • Lau J.
        Large trials vs meta-analysis of smaller trials.
        JAMA. 1996; 276: 1332-1338
        • Helfenstein U.
        • Steiner M.
        Fluoride varnishes (Duraphat).
        Community Dent Oral Epidemiol. 1994; 22: 1-5
        • Greenland S.
        Invited commentary.
        Am J Epidemiol. 1994; 140: 290-296
        • Pocock S.J.
        • Hughes M.D.
        Estimation issues in clinical trials and overviews.
        Stat Med. 1990; 9: 657-671
        • Sterling T.D.
        • Rosenbaum W.L.
        • Weinkam J.J.
        Publication decisions revisited.
        Am Stat. 1995; 49: 108-112
        • Chan W.C.
        • Fung S.C.
        Lung cancer in non-smokers in Hong Kong.
        in: Grundmann E. Cancer Campaign Vol 6. Cancer Epidemiology. Gustav Fischer Verlag, Stuttgart/New York1982: 199-202
        • Vandenbroucke J.P.
        Passive smoking and lung cancer.
        BMJ. 1988; 296: 391-392
        • Allison D.B.
        • et al.
        Publication bias in obesity treatment trials?.
        Int J Obes Relat Metab Disord. 1996; 20: 931-937
        • Begg C.B.
        • Mazumdar M.
        Operating characteristics of a rank correlation test for publication bias.
        Biometrics. 1994; 50: 1088-1101
        • Begg C.B.
        Publication bias.
        in: Cooper H. Hedges L.V. The Handbook of Research Synthesis. Russell Sage Foundation, New York1994: 399-409
        • Rust R.T.
        • Lehmann D.R.
        • Farley J.U.
        Estimating publication bias in meta-analysis.
        J Market Res. 1990; XXVII: 220-226
        • Persaud R.
        Misleading meta-analysis.
        BMJ. 1996; 312 ([Letter]): 125
        • Gleser L.J.
        • Olkin I.
        Models for estimating the number of unpublished studies.
        Stat Med. 1996; 15: 2493-2507
        • Hackshaw A.K.
        • Law M.R.
        • Wald N.J.
        The accumulated evidence on lung cancer and environmental tobacco smoke.
        BMJ. 1997; 315: 980-988
        • Sugita M.
        • Kanamori M.
        • Izuno T.
        • Miyakawa M.
        Estimating a summarized odds ratio whilst eliminating publication bias in meta-analysis.
        Jpn J Clin Oncol. 1992; 22: 354-358
        • Sugita M.
        • Yamaguchi N.
        • Izuno T.
        • Kanamori M.
        • Kasuga H.
        Publication probability of a study on odds ratio value circumstantial evidence for publication bias in medical study areas.
        Tokai J Exp Clin Med. 1994; 19: 29-37
        • Sugita M.
        • Izuno T.
        • Kanamori M.
        Recalculation of summarised odds ratios for the relationship between passive smoking and lung cancer based on data in the EPA report.
        Indoor Environment. 1995; 4: 177-181
      3. US Environmental Protection Agency. Respiratory health effects of passive smoking: lung cancer and other disorders. EPA/600/6-90-006F. Washington DC, 1992.

        • Easterbrook P.
        Reducing publication bias.
        BMJ. 1987; 295 ([Letter]): 1347
        • Simes R.J.
        Confronting publication bias.
        Stat Med. 1987; 6: 11-29
        • Smith R.
        Misconduct in research.
        BMJ. 1997; 315 ([Editorial]): 201-202
        • Naylor C.D.
        Meta-analysis and the meta-epidemiology of clinical research.
        BMJ. 1997; 315 ([Editorial]): 617-619
        • Berlin J.A.
        Will publication bias vanish in the age of online journals?.
        Online J Curr Clin Trials. 1992; Doc ([Editorial])
        • Irwig L.
        • Macaskill P.
        • Berry G.
        • Glasziou P.
        Bias in meta-analysis detected by a simple, graphical test. Graphical test is itself biased.
        BMJ. 1998; 316 ([Letter]): 470
        • Vandenbroucke J.P.
        Bias in meta-analysis detected by a simple, graphical test. Expert's views are still needed.
        BMJ. 1998; 316 ([Letter]): 469-470
        • Evans S.
        Misleading meta-analysis.
        BMJ. 1996; 312: 125