Chapter 7 Between-study Heterogeneity

By now, we have already shown you how to pool effect sizes in a meta-analysis. In meta-analytic pooling, we aim to synthesize the effects of many different studies into one single effect. However, this makes only sense if we aren’t comparing Apples and Oranges. For example, it could be the case that while the overall effect we calculate in the meta-analysis is small, there are still a few studies which report very high effect sizes. Such information is lost in the aggregate effect, but it is very important to know if all studies, or interventions, yield small effect sizes, or if there are exceptions.

It could also be the case that even some very extreme effect sizes were included in the meta-analysis, so-called outliers. Such outliers might have even distorted our overall effect, and it is important to know how our overall effect would have looked without them.

The extent to which effect sizes vary within a meta-analysis is called heterogeneity. It is very important to assess heterogeneity in meta-analyses, as high heterogeneity could be caused by between-studies differences. For example, a continuous variable, like the proportion of male participants, might influence the effect - or there might be two or more subgroups of studies present in the data, which have a different true effect. Such information could be very valuable for research, because this might allow us to find certain interventions or populations for which effects are lower or higher.

Very high heterogeneity could even mean that the studies have nothing in common, and that there is no “real” true effect behind our data, meaning that it makes no sense to report the pooled effect at all (Borenstein et al. 2011).

References

Borenstein, Michael, Larry V Hedges, Julian PT Higgins, and Hannah R Rothstein. 2011. Introduction to Meta-Analysis. John Wiley & Sons.