This function allows users to rely on the powerful caret package for cross-validating and tuning a rma analysis. Methods for rma are not included in the caret package, because the interface of caret is not entirely compatible with rma's model call. Specifically, rma is not compatible with the train methods for classes 'formula' or 'recipe'. The variance of the effect sizes can be passed to the 'weights' parameter of train.



ModelInfo list of length 13.


When using clustered data (effect sizes within studies), make sure to use 'index = groupKFold(your_study_id_variable, k = 10))' in traincontrol, to sample by study ID when creating cross-validation partitions; otherwise the testing error will be positively biased.


if (FALSE) { # Prepare data dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, dat$yi <- as.numeric(dat$yi) dat$alloc <- factor(dat$alloc) # Run rma rma.model <- rma(y = dat$yi, mods = dat[, c("ablat", "year")], vi = dat$vi) # R^2 is estimated to be .64 rma.model$R2 # Now, use cross-validation to see how well this model generalizes # Leave-one-out cross-validation is more appropriate than 10-fold cv because # the sample size is very small fit_control <- trainControl(method = "LOOCV") # Train the model without tuning, because rma has no tuning parameters <- train(y = dat$yi, x = dat[, c("ablat", "year")], weights = dat$vi, method = ModelInfo_rma(), trControl = fit_control) # Cross-validated R^2 is .08, suggesting substantial overfitting of the # original rma model$results$Rsquared }