LB 23-252 - Evaluating the precision gain from weighting in ecological meta-analyses

Background/Question/Methods:

In meta-analysis weighting observed effect sizes by the inverse of their variances to estimate the mean effect is known to produce the most precise estimates, and is often viewed as the optimal practice. However, the gain in precision by weighting has not been comprehensively evaluated for conditions representative of those found in ecological meta-analyses. We performed simulations of meta-analyses using log response ratio to compare the precision of the estimated mean effect size in weighted and unweighted analyses, for the random effects model, and a hierarchical extension. The simulations were done over a range of conditions observed in published ecological meta-analyses. Based on statistical theory we expected that precision gain of weighting would be influenced by 1) the heterogeneity of within-study variance (quantified by a new metric we derived) and 2) the relative magnitude of within- and among-study variance (quantified by the widely used I² metric). We evaluated whether these factors drove the precision gains of weighting. We also evaluated whether the confidence interval coverage for the weighted and unweighted analyses (which were constructed assuming either unequal or equal within study-variances, respectively) were approximately correct (i.e., did 95% confidence intervals cover the true mean effect about 95% of the time).

Results/Conclusions:

As expected, precision gain of weighting was influenced by 1) the heterogeneity of within-study variance and 2) the relative magnitude of within- and among-study variance (I²). Weighting provided more gain in precision when I² was low and heterogeneity was high. Weighting produced smaller gains in efficiency for the hierarchical model than for the random-effects model. Use of estimates of within-study variances instead of true values (even when estimates were imprecise) had little influence on the results. When I² was high (as is typical for ecological studies) precision gain of weighting was limited unless within-study variances were strongly heterogeneous. Nominal 95% confidence intervals had close to 95% coverage for both the weighted and unweighted analyses. Our results suggest that use of weighting is not always necessary and could be counterproductive in some cases: e.g., if weighting leads to bias (as it might if the magnitude of within-study variance depends on effect size), dropping of otherwise appropriate data, or use of limited software (and models). Our study provides a quantitative tool for analysts to estimate potential gain in precision from weighting and guide their choice of weighting in meta-analyses.