13.4 What to report

The preceding paragraphs offer a step-by-step instruction on how one might go about conducting a MetaForest analysis on a small sample meta-analytic dataset. One could simply apply these steps to a different dataset. However, reporting each step in detail might raise more questions than it answers, especially with readers and Reviewers unfamiliar with the machine learning approach in general, and MetaForest in specific. At the same time, it is essential that the analysis process is reproducible and transparent. This can be achieved by publishing the entire annotated syntax of the analysis as supplementary material, for example, on the Open Science Framework (www.osf.io). In fact, best practice would be to go one step further, and share the full data along with the syntax. In the text of the paper, one might then simply report a summary of the analysis, and provide a hyperlink to the DOI of the OSF page. The part of the results section describing the MetaForest analysis might read something like this:

“We conducted an exploratory search for relevant moderators using MetaForest: a machine-learning based approach to meta-analysis, using the random forests algorithm (Van Lissa 2017). Full syntax of this analysis is available on the Open Science Framework, DOI:10.17605/OSF.IO/XXXXX. To weed out irrelevant moderators, we used 100-fold replicated feature selection, and retained only moderators with positive variable importance in > 10% of replications. The main analysis consisted of 10.000 regression trees with fixed-effect weights, four candidate variables per split, and a minimum of three cases per terminal node. The final model had positive estimates of explained variance in new data, \(R^2_{oob} = \dots\), \(R^2_{cv} = \dots\). The relative importance of included moderators is displayed in Figure X. The shape of each moderator’s marginal relationship to the effect size, averaging over all values of all other moderators, is illustrated in Figure XX.”

---

References

Van Lissa, Caspar J. 2017. “MetaForest: Exploring Heterogeneity in Meta-Analysis Using Random Forests.” Open Science Framework, September. https://doi.org/10.17605/OSF.IO/KHJGB.