Chapter 10 Publication Bias

The file-drawer or publication bias problem is based on the fact that studies with big effect sizes are more likely to be published than studies with small effect sizes (Rothstein, Sutton, and Borenstein 2006).

The studies with low effect sizes might never get published, and therefore cannot be integrated into our meta-analysis. This leads to publication bias, as the pooled effect we estimate in our meta-analysis might be higher than the true effect size because we did not consider the missing studies with lower effects due to the simple fact that they were never published.

Although this practice is gradually changing (Nelson, Simmons, and Simonsohn 2018), whether a study is published still heavily depends on the statistical significance (\(p<0.05\)) of its results (Dickersin 2005). For any sample size, a result is more likely to become statistically significant if its effect size is high. This is particularly true for small studies, for which very large effect sizes are needed to attain a statisitcally significant result.

References

Dickersin, Kay. 2005. “Publication Bias: Recognizing the Problem, Understanding Its Origins and Scope, and Preventing Harm.” Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Wiley Chichester, UK, 11–33.

Nelson, Leif D, Joseph Simmons, and Uri Simonsohn. 2018. “Psychology’s Renaissance.” Annual Review of Psychology 69.

Rothstein, Hannah R, Alexander J Sutton, and Michael Borenstein. 2006. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. John Wiley & Sons.