Publication bias was not a good reason to discourage trials with low power

Abstract 

Objective

The objective was to investigate whether it is justified to discourage trials with less than 80% power. Trials with low power are unlikely to produce conclusive results, but their findings can be used by pooling then in a meta-analysis. However, such an analysis may be biased, because trials with low power are likely to have a nonsignificant result and are less likely to be published than trials with a statistically significant outcome.

Study Design and Setting

We simulated several series of studies with varying degrees of publication bias and then calculated the “real” one-sided type I error and the bias of meta-analyses with a “nominal” error rate (significance level) of 2.5%.

Results

In single trials, in which heterogeneity was set at zero, low, and high, the error rates were 2.3%, 4.7%, and 16.5%, respectively. In multiple trials with 80%–90% power and a publication rate of 90% when the results were nonsignificant, the error rates could be as high as 5.1%. When the power was 50% and the publication rate of non-significant results was 60%, the error rates did not exceed 5.3%, whereas the bias was at most 15% of the difference used in the power calculation.

Conclusion

The impact of publication bias does not warrant the exclusion of trials with 50% power.

Keywords: Meta-analysis, Publication bias, Ethics, Clinical trials, Type I error, Statistical power, heterogeneity