OOS 1-4 - Ecological meta-analysis is frequently wrong: quantifying the large and pervasive problem of non-independence

Background/Question/Methods
Meta-analysis is a powerful tool widely used by ecologists for synthesizing data. Ecologists have long been aware that ignoring non-independence in experimental work could lead to erroneous inferences. Similar challenges arise in meta-analysis (MA). Although methods exist to address relatively simple forms of non-independence in MA, a high percentage of current MAs, and an even higher percentage of older MAs, do not account for within paper non-independence. There is thus an implicit acceptance that such non-independence is of minimal concern with ecological data; however, there has been no formal examination of the magnitude of the problem. We applied a resampling approach to 20 recently published MAs to examine the robustness of their results, with special attention to issues associated with non-independence. For each published MA, we obtained the data and applied identical statistical approaches as in the original MA but added a categorical variable that randomly divided the source papers into two groups. We then evaluated if the difference between these two groups was significant with the expectation that only 5% of the comparisons would be significant. In a complimentary analysis, we also examined the influence of “nonsense” categorical variables associated with characteristics of each source article.

Results/Conclusions
The proportion of MAs showing significant effects (p < 0.05) for the random and nonsense categorical variables far exceeded the 5% expected for a valid analysis. For example, categorizing data by source article publication year as even or odd yielded a significant effect in 35% of the studies. Of the 20 MAs, those that accounted for some form of non-independence (7 MAs), such as geographic location, and those MAs that had, on average, low numbers of observations per source article, contributed less to deviation from the expectation for a valid analysis. For the original MAs that did not incorporate a random effect of paper (13 MAs) adding a random “paper effect” reduced, but did not eliminate, the problem of significance >5% of the time. Thus, when non-independence is ignored, MA falls short of the rigorous methodological advantage it is intended to have. Although including a random “paper effect” addresses a large portion of the problem in the MAs we examined, our results indicate that other, unaccounted for, sources of non-independence may exist. Therefore, ecologists should consider, and when possible, account for multiple sources of non-independence within their meta-analyses, and view with caution meta-analyses that do not.