ModelInfo_rma.RdThis function allows users to rely on the powerful caret package for
cross-validating and tuning a rma analysis. Methods for rma are not included
in the caret package, because the interface of caret is not entirely
compatible with rma's model call. Specifically, rma is not compatible with
the train methods for classes 'formula' or 'recipe'. The variance of
the effect sizes can be passed to the 'weights' parameter of train.
ModelInfo_rma()ModelInfo list of length 13.
When using clustered data (effect sizes within studies), make sure to use 'index = groupKFold(your_study_id_variable, k = 10))' in traincontrol, to sample by study ID when creating cross-validation partitions; otherwise the testing error will be positively biased.
if (FALSE) {
# Prepare data
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
dat$yi <- as.numeric(dat$yi)
dat$alloc <- factor(dat$alloc)
# Run rma
rma.model <- rma(y = dat$yi, mods = dat[, c("ablat", "year")], vi = dat$vi)
# R^2 is estimated to be .64
rma.model$R2
# Now, use cross-validation to see how well this model generalizes
# Leave-one-out cross-validation is more appropriate than 10-fold cv because
# the sample size is very small
fit_control <- trainControl(method = "LOOCV")
# Train the model without tuning, because rma has no tuning parameters
cv.mf.cluster <- train(y = dat$yi, x = dat[, c("ablat", "year")],
weights = dat$vi,
method = ModelInfo_rma(),
trControl = fit_control)
# Cross-validated R^2 is .08, suggesting substantial overfitting of the
# original rma model
cv.mf.cluster$results$Rsquared
}