Early extreme contradictory estimates may appear in published research: The Proteus phenomenon in molecular genetics research and randomized trials

Abstract 

Background and Objective

Divergent results on the same scientific question generate controversy. We hypothesized that controversial data are attractive to investigators and editors, and thus the most extreme, opposite results would appear very early rather than late, as data accumulate, provided data can be generated rapidly.

Methods

We used data from MEDLINE-indexed meta-analyses of case–control studies on genetic associations (retrospective, hypothesis-generating research with usually rapid turnaround) and meta-analyses of randomized trials of health care interventions (prospective, targeted research that usually takes longer) sampled from the Cochrane Library. Using cumulative meta-analysis, we evaluated how the between-study variance for studies on the same question changed over time and at what point the studies with the most extreme results ever observed had been published.

Results

The maximal between-study variance was more likely to be recorded early in the 44 eligible meta-analyses of genetic associations than in the 37 meta-analyses of health care interventions (P=.013). At the time of the first heterogeneity assessment, the most favorable-ever result in support of a specific association was more likely to appear than the least favorable-ever result (22 vs. 10, P=.017); the opposite was seen at the second heterogeneity assessment (15 vs. 5, P=.031). Such a sequence of extreme opposite results was not seen in the clinical trials meta-analyses. The estimated between-study variance decreased over time in genetic association studies (P=.010), but not in clinical trials (P=.30).

Conclusion

In contrast to prospective trials, a rapid early sequence of extreme, opposite results is frequent in retrospective hypothesis-generating molecular research.

Keywords: Bias, Meta-analysis, Publication bias, Genetics, Randomized controlled trials