The Bayesian interpretation of a P-value depends only weakly on statistical power in realistic situations

Abstract 

Objective

It is often repeated that a low P-value provides more persuasive evidence for a genuine effect if the power of the test is high. However, this is based on an argument which ignores the precise P-value in favor of simply observing whether P is less than some cut-off, and which oversimplifies the possible effect sizes. In a non-Bayesian framework, there are good reasons to think that power does not affect the evidence of a given P-value. Here I illustrate the relationship between pre-study power and the Bayesian interpretation of a P-value in realistic situations.

Study Design and Setting

A Bayesian calculation, using a conventional prior distribution for the effect size and a normal approximation to the sampling distribution of the sample estimate, where the datum is the precise P-value.

Results

Over the range of pre-study powers typical in published research, the Bayesian interpretation of a given P-value varies little with power.

Conclusion

A Bayesian analysis with reasonable assumptions produces results remarkably in line with a more simple, non-Bayesian intuition—that the evidence against the null hypothesis provided by a precise P-value should not depend on power.

Keywords: Bayes theorem, Hypothesis testing, Inference, P-value, Power of a test, Significance test