8.1 Mixed-Effects-Model

To conduct subgroup analyses using the Mixed-Effects-Model (random-effects-model within subgroups, fixed-effects-model between subgroups), you can simply include your grouping variable as a categorical predictor in the rma function. Like a classic t-test or ANOVA or regression model, this approach assumes homoscedasticity: The residual heterogeneity is assumed to be the same across groups.

We can use subgroup analysis to examine whether there are significant differences in the pooled effect between studies conducted within the USA, and studies conducted elsewhere. To do so, we first make a new variable, Country, that codes for studies conducted (partly) within the USA. Then, we specify the mixed-effects model with ~Country as a moderator. We drop the intercept by specifying -1, so the model estimates the mean effect size for both groups (as in ANOVA):

# Create a factor (categorical) variable for location
df$Country <- factor(df$location %in% c("USA", "USA/Korea"),
                     labels = c("Elsewhere", "USA"))

m_subgroup <- rma(yi = d, vi = vi, mods = ~ Country-1, data = df)
m_subgroup

## 
## Mixed-Effects Model (k = 56; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0503 (SE = 0.0164)
## tau (square root of estimated tau^2 value):             0.2243
## I^2 (residual heterogeneity / unaccounted variability): 64.66%
## H^2 (unaccounted variability / sampling variability):   2.83
## 
## Test for Residual Heterogeneity:
## QE(df = 54) = 138.1105, p-val < .0001
## 
## Test of Moderators (coefficients 1:2):
## QM(df = 2) = 39.6824, p-val < .0001
## 
## Model Results:
## 
##                   estimate      se    zval    pval   ci.lb   ci.ub 
## CountryElsewhere    0.1582  0.0572  2.7635  0.0057  0.0460  0.2704   ** 
## CountryUSA          0.3132  0.0553  5.6609  <.0001  0.2048  0.4216  *** 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

We see that the pooled effects of the subgroups differ quite substantially: \(g = .3132\) for studies conducted in the USA, and \(g = .1582\) for studies conducted elsewhere. But is the difference statistically significant? There are two ways we can find out.

8.1.1 Regression specification

We can test for the significance of the difference between groups by re-specifying the model using the regression specification: With an intercept, and an effect for the dummy variable Country, which is the difference between the two groups:

# Re-specify the model with an intercept and dummy
m_dummy <- rma(yi = d, vi = vi, mods = ~ Country, data = df)
m_dummy

## 
## Mixed-Effects Model (k = 56; tau^2 estimator: REML)
## 
## tau^2 (estimated amount of residual heterogeneity):     0.0503 (SE = 0.0164)
## tau (square root of estimated tau^2 value):             0.2243
## I^2 (residual heterogeneity / unaccounted variability): 64.66%
## H^2 (unaccounted variability / sampling variability):   2.83
## R^2 (amount of heterogeneity accounted for):            11.80%
## 
## Test for Residual Heterogeneity:
## QE(df = 54) = 138.1105, p-val < .0001
## 
## Test of Moderators (coefficient 2):
## QM(df = 1) = 3.7903, p-val = 0.0515
## 
## Model Results:
## 
##             estimate      se    zval    pval    ci.lb   ci.ub 
## intrcpt       0.1582  0.0572  2.7635  0.0057   0.0460  0.2704  ** 
## CountryUSA    0.1550  0.0796  1.9469  0.0515  -0.0010  0.3110   . 
## 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

The results indicate that the difference between the pooled effect sizes of studies conducted in the USA and elsewhere is \(\Delta g = 0.16, z = 1.95, p = .052\): Not significant.

8.1.2 T-test on the coefficients

Another way to test the significance of the difference is by manually conducting a post-hoc t-test on the two means from the model with ANOVA specification. The metaforest package contains a convenience function that conducts t-tests or z-tests on the parameters of rma meta-analyses. Now, we can use the function to conduct the t-test:

coef_test(m_subgroup, "CountryUSA", "CountryElsewhere")

## t(54) = 1.95, p = 0.06

The p-value is a bit higher, but otherwise the result is the same as the effect of the dummy variable in m_dummy. The difference in p-values is because we calculated a t-test here, whereas metafor uses z-tests by default.

8.1.3 Free variance per group

A logical question might be: Is it a reasonable assumption that the variance for studies conducted within the USA is equal to that of studies conducted elsewhere? After all - studies conducted elsewhere could come from all over the world, so they might be more heterogeneous. It is possible to free the variance between the subgroups, but this is quite advanced. There is an excellent tutorial on it online, if you need to do this in your future work:

http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates

---