7.2 Assessing the heterogeneity of your pooled effect size

Thankfully, once you’ve already pooled your effects in meta-analysis using the metafor() function, it is very easy and straightforward to retrieve the three most common heterogeneity measures that we described before.

In Chapter 5.2, we already showed you how to conduct a random-effect-model meta-analysis. In this example, we stored our results in the object m_re, which we will use again here.

One way to get heterogeneity measures of my meta-analysis is to print the meta-analysis (in my case, m_re) output again. Just running the name of an object calls the print() method for that object; notice that these two commands do the same thing:

m_re

## 
## Random-Effects Model (k = 56; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of total heterogeneity): 0.0570 (SE = 0.0176)
## tau (square root of estimated tau^2 value):      0.2388
## I^2 (total heterogeneity / total variability):   67.77%
## H^2 (total variability / sampling variability):  3.10
## 
## Test for Heterogeneity:
## Q(df = 55) = 156.9109, p-val < .0001
## 
## Model Results:
## 
## estimate      se    zval    pval   ci.lb   ci.ub 
##   0.2393  0.0414  5.7805  <.0001  0.1581  0.3204  *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

print(m_re)

## 
## Random-Effects Model (k = 56; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of total heterogeneity): 0.0570 (SE = 0.0176)
## tau (square root of estimated tau^2 value):      0.2388
## I^2 (total heterogeneity / total variability):   67.77%
## H^2 (total variability / sampling variability):  3.10
## 
## Test for Heterogeneity:
## Q(df = 55) = 156.9109, p-val < .0001
## 
## Model Results:
## 
## estimate      se    zval    pval   ci.lb   ci.ub 
##   0.2393  0.0414  5.7805  <.0001  0.1581  0.3204  *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We see that this output already provides us with all three heterogeneity measures (and even one more, H^2, which we will not cover here).

• \(\tau^{2}\), as we can see from the tau^2 output, is 0.0570.

• \(I^{2}\) is printed next to I^2, and has the value 67.77%.

• The value of \(Q\) is displayed next to Q under Test for heterogeneity:. As we can see, the value is 156.9109. In our case, this is significant (\(p < 0.001\); see p-value).

• We can also obtain a prediction interval, which we plotted in the forest plot, by running predict(m_re). As we can see, the 95% prediction interval (cr.lb and cr.ub) ranges from g=-0.2358 to 0.7143. It is calculated simply as \(\hat{\mu} \pm 1.96 * \hat{\tau}\): The estimated summary effect, with a 95% window based on the estimated standard deviation of the true effect.

How can we interpret the values of this example analysis? Well, all of our indicators suggest that moderate to substantial heterogeneity is present in our data. Given the broad prediction interval, which stretches well below zero, we also cannot be overly confident that the positive effect we found for our interventions is robust in every context. It might be very well possible that the intervention does not yield positive effects in some future scenarios; even a small negative effect might be possible based on the evidence the meta-analysis gives us. Very high effect sizes, on the other hand, are possible too.

---