Journal of Clinical Epidemiology
Volume 53, Issue 2 , Pages 207-216 , February 2000

Publication bias in meta-analysis: its causes and consequences

  • Alison Thornton

      Affiliations

    • P.N. Lee Statistics and Computing Ltd, 17 Cedar Road, Sutton, Surrey, SM2 5DA, United Kingdom
    • ,
  • Peter Lee

      Affiliations

    • P.N. Lee Statistics and Computing Ltd, 17 Cedar Road, Sutton, Surrey, SM2 5DA, United Kingdom
    • Corresponding Author InformationCorresponding author. Tel.: +44 181 642 8265; fax: +44 181 642 2135

Received 16 March 1998 ,Revised 21 June 1999 ,Accepted 26 July 1999.

References 

  1. In:  Last JM editors. A Dictionary of Epidemiology. Oxford: Oxford University Press; 1983;p. 12.
  2. Rosenthal R. The “file drawer problem” and tolerance for null results. Psychol Bull. 1979;86:638–641
  3. Smith MB. Editorial. J Abnorm Social Psychol. 1956;52:1–4
  4. Sterling T. Publication decisions and their possible effects on inferences drawn from tests of significance—or vice versa. Am Stat Assoc J. 1959;54:30–34
  5. Smart RG. The importance of negative results in psychological research. Can Psychol. 1964;5:225–232
  6. Tweedie RL, Scott DJ, Biggerstaff BJ, Mengersen KL. Bayesian meta-analysis, with application to studies of ETS and lung cancer. Lung Cancer. 1996;14(suppl. 1):S171–S194
  7. Angell M. Negative studies. N Engl J Med. 1989;321:464–466
  8. Newcombe RG. Towards a reduction in publication bias. BMJ. 1987;295:656–659
  9. Pocock SJ, Hughes MD, Lee RJ. Statistical problems in the reporting of clinical trials (a survey of three medical journals). N Engl J Med. 1987;317:426–432
  10. Green SJ, Fleming TR, Emerson S. Effects on overviews of early stopping rules for clinical trials. Stat Med. 1987;6:361–369
  11. Felson DT. Bias in meta-analytic research. J Clin Epidemiol. 1992;45:885–892
  12. Lee PN. Problems in interpreting epidemiological data. In:  Mohr U,  Bates DV,  Dungworth DL,  Lee PN,  McClellan RO,  Roe FJC editor. Assessment of Inhalation Hazards. Berlin: Springer-Verlag; 1989;p. 49–50
  13. Begg CB, Berlin JA. Publication bias (a problem in interpreting medical data). J R Stat Soc A. 1988;151:419–463
  14. Givens GH, Smith DD, Tweedie RL. Publication bias in meta-analysis (A Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate). Stat Sci. 1997;12:221–250
  15. Engler RI, Covell JW, Friedman PJ, Kitcher PS, Peters RM. Misrepresentation and responsibility in medical research. N Engl J Med. 1983;317:1383–1389
  16. Rosenthal R, Fode KL. Psychology of the scientist (V. Three experiments in experimenter bias). Psychol Rep. 1963;12:491–511
  17. Rosenthal R. Experimenter outcome-orientation and the results of the psychological experiment. Psychol Bull. 1964;61:405–412
  18. Chalmers TC, Berrier J, Sacks HS, Levin H, Reitman D, Nagalingam R. Meta-analysis of clinical trials as a scientific discipline. I (Control of bias and comparison with large co-operative trials). Stat Med. 1987;6:315–328
  19. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA. 1990;263:1385–1389
  20. Kleijnen J, Knipschild P. Review articles and publication bias. Arzneimittelforschung. 1992;42:587–591
  21. Minerva . Re publication bias. BMJ. 1992;304:264
  22. Light RJ. Accumulating evidence from independent studies (what we can win and what we can lose). Stat Med. 1987;6:221–231
  23. Begg CB. A measure to aid in the interpretation of published clinical trials. Stat Med. 1985;4:1–9
  24. Bulpitt CJ. Meta-analysis. Lancet. 1988;ii:93–94
  25. Koren G, Shear H, Graham K, Einarson T. Bias against the null hypothesis (the reproductive hazards of cocaine). Lancet. 1989;ii:1440–1442
  26. Chalmers TC, Frank CS, Reitman D. Minimizing the three stages of publication bias. JAMA. 1990;263:1392–1395
  27. Charlton B. Think negative (science needs its failures). New Scientist. 1987;29Oct:72
  28. Mahoney MJ. Publication prejudices (an experimental study of confirmatory bias in the peer review system). Cog Ther Res. 1977;1:161–175
  29. LeVois ME, Layard MW. Publication bias in the environmental tobacco smoke/coronary heart disease epidemiologic literature. Regul Toxicol Pharmacol. 1995;21:184–191
  30. Bero LA, Galbraith A, Rennie D. Sponsored symposia on environmental tobacco smoke. JAMA. 1994;271:612–617
  31. L'Abbé KA, Detsky AS, O'Rourke K. Meta-analysis in clinical research. Ann Intern Med. 1987;107:224–233
  32. Mosteller F, Colditz GA. Understanding research synthesis (meta-analysis). Annu Rev Public Health. 1996;17:1–23
  33. Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet. 1991;337:867–872
  34. Blair A, Burg J, Foran J, Gibb H, Greenland S, Morris R, et al. Guidelines for application of meta-analysis in environmental epidemiology. Regul Toxicol Pharmacol. 1995;22:189–197
  35. Greenland S. Quantitative methods in the review of epidemiologic literature. Epidemiol Rev. 1987;9:1–30
  36. Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. N Engl J Med. 1987;316:450–455
  37. Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A, McIlroy W, Oxman AD. Should unpublished data be included in meta-analyses? JAMA 1993;269:2749–53.
  38. Kilpatrick SJ. The epidemiology of environmental tobacco smoke (ETS) and the weight of evidence argument. Int Surg. 1992;77:131–133
  39. Pignon J-P, Arriagada R. Meta-analyses of randomized clinical trials (how to improve their quality?). Lung Cancer. 1994;10(suppl. 1):S135–S141
  40. Wald NJ, Nanchahal K, Thompson SG, Cuckle HS. Does breathing other people's tobacco smoke cause lung cancer?. BMJ. 1986;293:1217–1222
  41. Jones DR. Meta-analysis of observational epidemiological studies (a review). J R Soc Med. 1992;85:165–168
  42. DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clin Trials. 1986;7:177–188
  43. Moher D, Fortin P, Jadad AR, Jüni P, Klassen T, Le Lorier J, et al. Completeness of reporting trials published in languages other than English (implications for conduct and reporting of systematic reviews). Lancet. 1996;347:363–366
  44. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629–634
  45. Grégoire G, Derderian F, Le Lorier J. Selecting the language of the publications included in a meta-analysis (is there a Tower of Babel bias?). J Clin Epidemiol. 1995;48:159–163
  46. Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Controlled Clin Trials. 1998;19:159–166
  47. Gøtzsche PC. Reference bias in reports of drug trials. BMJ. 1987;295:654–656
  48. Tramèr MR, Reynolds DJM, Moore RA, McQuay HJ. Impact of covert duplicate publication on meta-analysis (a case study). BMJ. 1997;315:635–640
  49. Morris RD. Meta-analysis in cancer epidemiology. Environ Health Perspect. 1994;102(suppl. 8):61–66
  50. Cappelleri JC, Ioannidis JPA, Schmid CH, de Ferranti SD, Aubert M, Chalmers TC, et al. Large trials vs meta-analysis of smaller trials (how do their results compare?). JAMA. 1996;276:1332–1338
  51. Helfenstein U, Steiner M. Fluoride varnishes (Duraphat) (a meta-analysis). Community Dent Oral Epidemiol. 1994;22:1–5
  52. Greenland S. Invited commentary (a critical look at some popular meta-analytic methods). Am J Epidemiol. 1994;140:290–296
  53. Pocock SJ, Hughes MD. Estimation issues in clinical trials and overviews. Stat Med. 1990;9:657–671
  54. Sterling TD, Rosenbaum WL, Weinkam JJ. Publication decisions revisited (the effect of the outcome of statistical tests on the decision to publish and vice versa). Am Stat. 1995;49:108–112
  55. Chan WC, Fung SC. Lung cancer in non-smokers in Hong Kong. In:  Grundmann E editors. Cancer Campaign Vol 6. Cancer Epidemiology. Stuttgart/New York: Gustav Fischer Verlag; 1982;p. 199–202
  56. Vandenbroucke JP. Passive smoking and lung cancer (a publication bias?). BMJ. 1988;296:391–392
  57. Allison DB, et al.  Publication bias in obesity treatment trials?. Int J Obes Relat Metab Disord. 1996;20:931–937
  58. Begg CB, Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics. 1994;50:1088–1101
  59. Begg CB. Publication bias. In:  Cooper H,  Hedges LV editor. The Handbook of Research Synthesis. New York: Russell Sage Foundation; 1994;p. 399–409
  60. Rust RT, Lehmann DR, Farley JU. Estimating publication bias in meta-analysis. J Market Res. 1990;XXVII:220–226
  61. Persaud R. Misleading meta-analysis (“fail-safe N” is a useful mathematical measure of the stability of results). BMJ. 1996;312:125;[Letter]
  62. Gleser LJ, Olkin I. Models for estimating the number of unpublished studies. Stat Med. 1996;15:2493–2507
  63. Hackshaw AK, Law MR, Wald NJ. The accumulated evidence on lung cancer and environmental tobacco smoke. BMJ. 1997;315:980–988
  64. Sugita M, Kanamori M, Izuno T, Miyakawa M. Estimating a summarized odds ratio whilst eliminating publication bias in meta-analysis. Jpn J Clin Oncol. 1992;22:354–358
  65. Sugita M, Yamaguchi N, Izuno T, Kanamori M, Kasuga H. Publication probability of a study on odds ratio value circumstantial evidence for publication bias in medical study areas. Tokai J Exp Clin Med. 1994;19:29–37
  66. Sugita M, Izuno T, Kanamori M. Recalculation of summarised odds ratios for the relationship between passive smoking and lung cancer based on data in the EPA report. Indoor Environment. 1995;4:177–181
  67. US Environmental Protection Agency. Respiratory health effects of passive smoking: lung cancer and other disorders. EPA/600/6-90-006F. Washington DC, 1992.
  68. Easterbrook P. Reducing publication bias. BMJ. 1987;295:1347;[Letter]
  69. Simes RJ. Confronting publication bias (a cohort design for meta-analysis). Stat Med. 1987;6:11–29
  70. Smith R. Misconduct in research (editors respond. The Committee on Publication Ethics (COPE) is formed). BMJ. 1997;315:201–202[Editorial]
  71. Naylor CD. Meta-analysis and the meta-epidemiology of clinical research. BMJ. 1997;315:617–619[Editorial]
  72. Berlin JA. Will publication bias vanish in the age of online journals?. Online J Curr Clin Trials. 1992;Doc(No 12):[Editorial]
  73. Irwig L, Macaskill P, Berry G, Glasziou P. Bias in meta-analysis detected by a simple, graphical test. Graphical test is itself biased. BMJ. 1998;316:470;[Letter]
  74. Vandenbroucke JP. Bias in meta-analysis detected by a simple, graphical test. Expert's views are still needed. BMJ. 1998;316:469–470[Letter]
  75. Evans S. Misleading meta-analysis (statistician's comment). BMJ. 1996;312:125

PII: S0895-4356(99)00161-4

Journal of Clinical Epidemiology
Volume 53, Issue 2 , Pages 207-216 , February 2000

Article Tools

* Image Usage & Resolution