Generalizing ecological relationships over space and time often is beyond the scope of any one research project, meaning that summarizing the results of many studies is important for gaining understanding of ecology. One challenge when collectively analyzing many studies is heterogeneity of study results. Meta-analysis may be ill-fit for analyzing the results of studies that are too different to be compared. Standardization of methodologies may be a solution to the problem of heterogeneity. The Nutrient Network (NutNet) is a coordinated research effort carrying out a distributed experiment in 130 grassland sites around the world, using standardized methods. The experiment tests effects of nutrient enrichment, mammalian herbivore removal, and their interaction on aboveground plant biomass. Here we ask whether NutNet’s methodological standardization reduces heterogeneity of results across sites, relative to comparable meta-analyses. We also ask whether NutNet’s methodological standardization improves its power to detect effects of ecological covariates such as latitude and rainfall, relative to comparable meta-analyses. We do this by fitting mixed effect models to data from NutNet and a comparable meta-analysis, and testing the effects of ecological covariates. We also compare the among-site variance in effect size in NutNet to the among-study variance in effect size of comparable meta-analyses.
Results/Conclusions
Preliminary results indicate that effects of nutrient enrichment and herbivore removal are heterogeneous among NutNet sites; ~70-80% of among-site variance in effect size is attributable to heterogeneity (among-site variation in true mean effect size). In typical ecological meta-analyses, ~90% of variance in effect size is due to heterogeneity (Senior et al. 2016). Meta-analyses of the effects of herbivore removal on plant biomass exhibit similar heterogeneity to NutNet (Jia et al. 2018). Further, in meta-analyses, among-study variance in effects of herbivore removal and nutrient enrichment is comparable to among-site variance in effects in NutNet (Jia et al. 2018, Gruner et al. 2007). Ecological covariates explain as much as 13% of the heterogeneity in effect size in NutNet, more than comparable meta-analyses, but this comparison is difficult to interpret due to lack of data on some covariates. The similar heterogeneity of NutNet and meta-analyses of herbivore removal is surprising, given that different studies in meta-analyses involve different species, habitats, and methods. Our results suggest that methodological differences are not a large source of heterogeneity in study results. Perhaps methodological consistency aids less in reducing heterogeneity and more in making heterogeneity partially explainable with moderator variables that describe the narrowed-down differences among sites.
