First, we separately conducted systematic reviews of the literature with the goal of finding all published primary sources on ecological outcomes of Tamarix control in the American Southwest. We conducted a literature search in October 2019 using the following search terms: “(Tamarix or tamarisk* or saltcedar) and (restor* or remov* or biocontrol or Diorhabda) and (river or riparian or floodplain or stream)”, filtered by “Article” in Web of Science. To provide a second set of starting sources (specifically seeking non-journal sources in addition to journal sources), we then conducted a search in March 2020 using the following search terms: (tamarisk or Tamarix or “salt cedar” or saltcedar) and (remov* or (invasive* and (control* or manag*))), filtered by “Article” in the following databases: Aquatic Sciences and Fisheries Abstracts, ProQuest Agricultural and Environmental Science Collection, Academic Search Complete, Biological Abstracts, GreenFILE and Web of Science Core Collection. The March 2020 search yielded an initial total of 1,320 articles, the October 2019 search yielded 266 and the February 2021 search yielded 42. In addition, we manually added 15 sources not found in the literature searches, based on professional judgement of their fit with the goals of the study and finally conducted another identical database search in February 2021 to identify newly-published literature since the March 2020 search; four new sources were added as a result of this search. Peer-reviewed published articles, doctoral dissertations and government reports were ultimately included as sources.

Initial filtering (duplicates, title and abstract relevance) happened separately for each search. All filtering at this stage was based on the same criteria; sources were included if they investigated some aspect of an ecological outcome of active or biological control of Tamarix spp. in North America, which automatically also narrowed the papers to only those with a reported focus on T. ramosissima and/or T. chinensis and its hybrids. We first filtered out duplicate sources in each search, then filtered the resulting list based on title; papers excluded in this first round were mostly concerned with other aspects of Tamarix biology and ecology not related to control. The March 2020 search had a high number of duplicate sources (n = 758) due to searching multiple databases. Very few papers (n = 9) were excluded solely due to research taking place outside North America; other research conducted outside North America studied Tamarix in its native range or was not related to ecology. Following removal of duplicates between the two searches, this step yielded a total of 109 sources that we read in their entirety, subsequently filtered to 81, based on full text content. Papers were excluded at this stage if they did not address intentional anthropogenic treatment of Tamarix or only involved greenhouse studies without in situ field data. Papers from the final filtering stage were combined with the sources we manually selected, for a final sample size of 96 sources (Fig. 1A). We then conducted the three tiers of analysis, with increasing restrictions, based on what quantitative measures were included (Fig. 1B). This was done also to investigate the hypothesis that a traditional meta-analysis could bias the findings, a common criticism of a strict quantitative approach (Gurevitch and Hedges 1999; Koricheva and Gurevitch 2014; Haddaway et al. 2015; Westgate and Lindenmayer 2017; Lilian et al. 2021).

Figure 1.

A summary of searches and source filtering B summary of the three tiers of analysis.

For inclusion in any of the three tiers of analysis, these papers needed to explicitly address active control of Tamarix and/or biocontrol and include some measure of the effect of treatment on an ecosystem component (either measurement before and after treatment [“BA”] or a control group compared to an impact group [“CI”]); of these, 96 papers were ultimately selected for use in tier 1: tracing Tamarix control evaluation, 63 for tier 2: qualitative vote count and 52 for tier 3: meta-analysis. Criteria for including or excluding papers for tiers 2 and 3 are described in detail below. While searches included multiple databases, all papers ultimately selected (including dissertations and agency reports) were catalogued in Web of Science. Refer to Suppl. material 2 for a complete list of publications.

For each paper, we recorded sampling location data (river basin; Upper Colorado River Basin, Lower Colorado River Basin, Rio Grande River Basin and Humboldt River Basin), study design, control and/or restoration actions (using definitions outlined in González et al. (2017b); Table 1), response variables (Table 2) and study duration (first year of restoration, last year of restoration, first year of monitoring and last year of monitoring; Morandi et al. (2014); González et al. (2015)). Each measure of an ecosystem component (e.g. native plant species richness following biocontrol) was recorded separately as a row of data. Within a paper, we designated separate “case IDs”, based on whether multiple rows of data were independent replicates of each other; for example, if a source reported multiple results of the same test in two separate sites, they would be considered distinct cases, whereas multiple replicates at one site would be designated as the same case.

Tracing trends in the literature is important to understand how priorities and approaches have changed over time and to identify knowledge gaps (Dufour et al. 2019), particularly since restoration ecology is a relatively young discipline and its protocols are still subject to further development (Hobbs 2018). It also has fewer constraints than quantitative analysis, allowing us to include all publications that investigated response to restoration in areas with Tamarix invasion (n 96; Fig. 1). We summarised basic trends in study foci over time and conducted a qualitative success ranking based on publication abstracts to assess messages in the literature at coarser resolution.

For each selected source, restoration treatments and measured ecosystem component responses to restoration were categorised (e.g. plants, water, invertebrates) to determine relative numbers of different abiotic and biotic characteristics addressed in restoration studies (Table 2). For the purpose of this study, we considered “restoration treatments” to include all types of intentional control of invasive Tamarix (e.g. chemical, mechanical, biological control) as well as follow-up actions, such as revegetation of desirable species or follow-up herbicide application. Categories were adapted from those described in Gann et al. (2019), Ruiz-Jaen and Aide (2005), Morandi et al. (2014) and González et al. (2015) and combined where necessary.

We then conducted a qualitative review of “success” of Tamarix control reported in the abstracts of each of the 96 papers, using a scale of 1 to 5, plus a category for those that appeared inconclusive (Table 3). These scores were intended to reflect the degree of positive result from restoration, as indicated by the papers themselves, such as whether outcomes were consistent with goals (e.g. reduction in invasive species abundance or increase in desirable species abundance). In other words, the published opinion of “success” could have been supported with numerical data or simply reported as a subjective observation for this tier of our literature review. Each paper was assigned a numerical value of “success” by averaging scores assigned to it by four of the authors (AG, EG, AS, AH) in a blind review (i.e. scorers did not view each other’s scores until afterwards). Most papers were already familiar to us; due to time constraints, we made our success categorisations based primarily on the abstracts, but reviewed the discussion/conclusions and introduction if needed for additional context. If scorers disagreed on whether a paper was inconclusive or not, the majority opinion was accepted and, in the case of a tie, we revisited the abstract and made a decision. To help detect any bias in paper selection, based on the three tiers of analysis, we also compared the breakdown of success rankings amongst the three pools of papers (i.e. all 96 papers evaluated in tracing, the subset of 63 papers evaluated in the vote count and the subset of 52 papers evaluated in the meta-analysis). We averaged success rankings by scorer for each paper and counted the number of papers whose average ranking fell within each range.

For identifying quantitative directional trends with the largest possible sample size, we then conducted vote counts of general outcomes for the 63 sources that included quantitative measures of impact of Tamarix control (Fig. 1). All sources included in the vote counts were also included in the success ranking section. Response metrics differed in terms of desirability (e.g. sources may report changes in Tamarix abundance as well as abundance of native species). As our goal was to make direct comparisons amongst all methods, it was necessary to standardise the directionality to always refer to desired outcomes. Therefore, each row of data was designated either desirable, undesirable or neutral for the purpose of calculating effect sizes with consistent directionality (i.e. the dependent variable in all cases is “improvement,” which could consist of either reduction in undesirable environmental characteristics or increase in desirable characteristics). Desirability was categorised on the basis of stated goals of each project and the general assumption that higher native plant and animal abundance, as well as overall higher species diversity, is desired, while higher invasive plant abundance is not desired (Shafroth et al. 2008). Thus, we counted non-noxious exotic species (as defined by USDA PLANTS; USDA, NRCS (2024)) as desirable since they contribute to biodiversity. Data reported in graph form were digitised manually to the highest possible accuracy using GraphGrabber v.2.02 (Quintessa Inc.). All records were checked by someone other than the coder at least once.

The first vote count tallied all outcomes of Tamarix control that were reported as statistically significant (n = 622 outcomes). As not all cases within a paper reported results of an associated statistical test (for example, if results simply showed a list of before and after values for cover of multiple species), we then conducted a separate vote count across the entire dataset, regardless of significance (n = 1,460 outcomes). All ecosystem components were assigned a vote count value, based on whether the response variable significantly increased, decreased or did not significantly change over time (for before-after comparisons; abbr. BA) or between the control and impact groups (abbr. CI). When a case was reported as a BACI design (Before-After-Control-Impact; reporting before-after data for both the control group and the treatment group), we split it into separate BA and CI cases (or rows in the database). We also recorded effect size regardless of statistical significance. We then calculated relative percentages of increased/decreased/no change metrics for each possible combination of restoration treatment and ecosystem component.

Finally, we conducted a meta-analysis to statistically test the hypothesis that Tamarix control activities resulted in positive outcomes (Shafroth et al. 2008), as measured in terms of various biotic and abiotic factors (n = 52, Tables 1, 2). Mean, sample size and variance were required for a measurement to be included in the meta-analysis, making it the synthesis method with the most stringent requirements. We used the metafor package in R (Viechtbauer 2010) to calculate effect sizes of each case, represented as standardised mean differences (Viechtbauer 2010). To standardise directionality of metrics, based on desirability (e.g. increases in native species cover were considered desirable, but so were decreases in invasive species cover), we multiplied the effect size by –1 when a response metric was considered undesirable and response metrics designated as neutral were excluded from analysis. We added a small constant (0.000001) to all standard deviations in order to allow for calculation of effect sizes in cases where the variance was zero. We then constructed separate multi-level error models for each possible combination of restoration actions and ecosystem components (e.g. all vegetation responses to biocontrol). A restoration action was included in each model as a moderator and the effect size of each action on each ecosystem component was used as a dependent variable. Case ID, nested within paper ID (unique identifier for each paper), was included as a random effect in all models. In many cases, there was insufficient replication (fewer than three replicates) to subset the data by a given combination of response metric and restoration action; we did not report an effect size for these subsets. In addition, some categories (e.g. fauna subcategories) were combined to improve replication, as sample sizes were lower in the meta-analysis than in the vote counts.

In addition to calculating effect sizes by treatment and response variable, we conducted a sub-study using all restoration actions and vegetation (the most well-represented ecosystem component) divided by desirability category (desirable, Tamarix and undesirable other than Tamarix) and growth habit (understorey, overstorey, both). We also examined the impact of temporal scale on vegetation outcomes, using the following metrics for elapsed time: (1) number of years between end of treatment and start of monitoring; (2) number of years between end of treatment and end of monitoring.

We tested effects of various characteristics of restoration projects (duration and geographic location, by river basin) to determine whether they affect the effect sizes. In some cases, there were few to no between-paper replicates of a specific restoration action/response metric combination; we report both number of papers addressing each metric and number of discrete measurements of each restoration action/ecosystem component combination. We also tested for funnel plot asymmetry using Egger’s test (Egger et al. 1997) and calculated fail-safe N values using the Rosenthal method (Rosenthal 1979; Orwin 1983) for each model to determine whether significant results were being influenced by insufficient sample sizes (Viechtbauer 2010). Three data points were excluded from analysis due to extremely high variance.

All analyses were conducted in R version 4.1.1 with RStudio version 2022.02.03 using the following functions from the metafor package: “escalc” (calculates effect sizes from means, SDs, Ns), “rma.mv” (mixed model calculation), “fsn” (calculates fail-safe N value) and “funnel” (creates funnel plots to visualise asymmetry; Viechtbauer (2010)).

Publication trends by year

The bulk of papers on the effect(s) of Tamarix control were published between 2011 and 2020 and the largest number (10) were published in 2017. However, there were no clear directional trends over time. Most of the papers included in our analysis focused on vegetation metrics (78% of reported outcomes were on vegetation; Suppl. material 1: table S3), with particular years featuring more measurements of fauna (2015) or abiotic responses (2017). There were transitions over time with regard to treatment methods, with the cut stump method appearing less often, as heavy machinery and biocontrol became more common around 2011 and, ultimately, were the two best-represented treatment methods investigated (Suppl. material 1: table S4). However, it is notable that, like other measures, publications reporting response to biocontrol did not increase over time, instead having peaks in 2011 and 2020.

Success rankings

Our success rankings showed that outcomes reported in paper abstracts were, on average, slightly positive, i.e. between “neutral” (no effects or some positive effects on some components compensate negative effects on others) and “partial success” (the message is positive, but there is a “however”); mean = 3.61; median = 3.75; SD = 0.99. Similarly, the counts of averaged success rankings show a high proportion of abstracts reporting partially successful outcomes (Fig. 2). Agreement between scorers (AG, AS, EG, AH) was high; we had perfect agreement on 29 of 96 papers and near-perfect agreement (three out of four scorers agreed and the fourth ranking was an adjacent value) on 45 of 96 papers. We did not find a difference amongst the distributions of success rankings across the full 96 papers, the 63-paper vote count subset and the 52-paper meta-analysis subset (ANOVA: F = 1.23, p = 0.30, df = 2).

Figure 2.

Distribution showing counts of averages of scorers’ rankings for “success” of projects as inferred from language in each publication abstract. “Success” was considered in relation to the stated goals of the study (if present). Outcomes described in papers were considered “inconclusive” if the majority of scorers reported that the authors of the paper discussed Tamarix control, but focused on methods rather than outcomes. n = 96. See Table 3 for definitions of success categories.

The vote counts found that most vegetation responses to Tamarix control efforts showed more positive than negative outcomes (Fig. 3; blue predominated in stacked bar charts of first row), but “no change” predominated when only examining reported statistically significant changes (Fig. 4; grey predominated in stacked bar charts of first row). Sample size of fauna outcomes in response to Tamarix control efforts was very low (two or fewer publications per combination of treatment method and response metric), but showed relatively high numbers of negative outcomes; birds showed the most negative outcomes (negatively affected by biocontrol and cut-stump with herbicide; Figs 3, 4, second row) and herpetofauna were negatively affected by biocontrol in all cases (statistically significantly in half of cases), but showed generally positive or neutral outcomes from other treatment methods. Abiotic results were mixed; there were more positive “water” outcomes (primarily reductions in evapotranspiration) than negative (Fig. 3, rows 7–12), but more negative hydrological outcomes, and geomorphic outcomes were almost entirely value-neutral (Figs 3, 4, final two rows).

Figure 3.

Summary of vote counting by treatment method used to control Tamarix, based on whether any change at all was reported in the publication. Each cell represents a combination of the listed treatment method and response variable. Bars represent the numbers of desirable outcomes (shown in blue), undesirable outcomes (red), neutral outcomes (dark grey) and no-change (light grey). Width reflects sample size, with number of observations (number of papers in parentheses) reported in each cell.

Figure 4.

Summary of vote counting by treatment method used to control Tamarix, only if change was reported as statistically significant by the published source. Each cell represents a combination of the listed treatment method and response variable. Bars represent the numbers of desirable outcomes (shown in blue), undesirable outcomes (red), neutral outcomes (dark grey) and no-change (light grey). Width reflects sample size, with number of observations (number of papers in parentheses) reported in each cell.

The vegetation-only vote count on differences regardless of statistical significance showed broadly that Tamarix cover was reduced in most cases and non-Tamarix undesirable vegetation was heavily reduced in the overstorey, but not the understorey (Fig. 5). Response of desirable vegetation was mixed, with slightly better outcomes in the overstorey than the understorey (Fig. 5). The vegetation-specific vote count on significant differences found that “no change” was very common, but reduction in overall Tamarix metrics and non-Tamarix invasive overstorey species were seen in most cases (Fig. 6). Changes in desirable vegetation were mixed, but there were more positive than negative outcomes of total desirable vegetation cases.

Figure 5.

Summary of vote counting by vegetation types, change in vegetation in response to Tamarix control efforts regardless of reported statistical significance in the published paper. Each cell represents a combination of invasive classification (desirable/undesirable/total) and growth habit (overstorey/understorey/both). Bars represent the numbers of desirable outcomes (shown in blue), undesirable outcomes (red), neutral outcomes (dark grey) and no-change (light grey). Width reflects sample size, with number of observations (number of papers in parentheses) reported in each cell. Note that “overstorey” and “understorey” sample sizes do not add up to “overstorey + understorey” sample sizes, as response variables were not always reported as specific overstorey/understorey metrics.

Figure 6.

Summary of vote counting by vegetation types, only if change in vegetation type in response to Tamarix control effort was reported as statistically significant by the published source. Each cell represents a combination of invasive classification (desirable/undesirable/total) and growth habit (overstorey/understorey/both). Bars represent the numbers of desirable outcomes (shown in blue), undesirable outcomes (red), neutral outcomes (dark grey) and no-change (light grey). Widths reflect sample size, with number of observations (number of papers in parentheses) reported in each cell. Note that “overstorey” and “understorey” sample sizes do not add up to “overstorey + understorey” sample sizes, as response variables were not always reported as specific overstorey/understorey metrics.

Total sample size for the meta-analysis was 777 outcomes within 52 publications. The overall model without considering any moderator was heterogeneous (Q(df = 771) = 9,238 p < 0.0001)) and there was a significant, but small positive effect of Tamarix control (estimated effect size = 0.5465, SE = 0.2732, Z = 2.0002, p = 0.045). The fail-safe N calculation on effect sizes via the Rosenthal method was significant (p < 0.0001), with a fail-safe N of 409,193.

By treatment method

Most of the significant effects of treatments on ecosystem components were positive (treatments were associated with more desirable outcomes). Restoration treatments were broadly seen to either decrease cover of undesirable plant species, increase cover of desirable plant species or have no effect; herbicide had the highest significant positive effect on desirable outcomes. Amongst the 19 response variables, we found the effect sizes of six to be significantly different from zero, including the following combinations with treatments: biocontrol, cut-stump with herbicide, herbicide and cutting were associated with positive vegetation outcomes (biocontrol: est. = 0.3985, Z = 1.98, p < 0.05; cut-stump: est. = 0.26, Z = 2.12, p < 0.05, herbicide: est. = 1.30, Z = 3.70, p < 0.001; cutting: est. = 0.20, Z = 4.72, p < 0.0001), cut-stump treatment was associated with positive water outcomes (est. = 0.656, Z = 2.26, p = 0.02) and herbicide was associated with negative fire outcomes (est. = -0.333, Z = -3.44, p < 0.001; Fig. 7). Some response variable categories are condensed in the meta-analysis relative to the vote counts due to lower sample sizes (e.g. while the vote counts differentiate between birds, fish, mammals and herpetofauna, the meta-analysis combines all fauna).

Figure 7.

Summary of quantitative meta-analysis examining responses of multiple ecosystem components to control of Tamarix by multiple methods as reported in published papers. Dots represent the effect size estimate, calculated as the standardised mean difference. Horizontal lines represent 95% confidence intervals and vertical dotted lines denote zero. Asterisks next to dots indicate statistical significance; sample sizes are shown next to dots with number of studies reported in parentheses. Blue dots represent significantly positive effect sizes and red dots represent significantly negative effect sizes.

Vegetation by growth habit and category

Total vegetation (including desirable, Tamarix and other undesirable; Fig. 8) showed statistically significant positive responses to treatment across growth habits (overstorey+understorey: Z = 2.40, p < 0.05, k = 607; overstorey: Z = 2.19, p < 0.05, k = 97; understorey: Z = 2.35, p ≤ 0.05, k = 191). However, this result appears to be primarily driven by the significant response of Tamarix reduction in the overstorey+understorey (Z = 3.02, p < 0.01, k = 275), as cover of desirable and non-Tamarix undesirable vegetation was not observed to change significantly as a result of Tamarix control efforts.

Figure 8.

Summary of quantitative meta-analysis of vegetation-only data from published sources (for all treatment methods used to control Tamarix) by vegetation types (understorey/overstorey/both, desirable/undesirable/all). Dots represent the effect size estimate, calculated as the standardised mean difference. Horizontal lines represent 95% confidence intervals and vertical dotted lines denote zero. Asterisks next to dots indicate statistical significance; sample sizes are shown next to dots with number of studies reported in parentheses. Blue dots represent significantly positive effect sizes.

Our results indicate that while Tamarix is successfully reduced by control efforts, other ecosystem components are less clearly affected and, in some cases, are negatively impacted. When examining all vegetation metrics, we see a generally positive effect that is mostly being driven by reduction of Tamarix in both the understorey and overstorey. While most effect sizes for native vegetation metrics were non-significant (and all were small in magnitude, with high variance), all were positive. This is consistent with previous research showing that increases in native cover following Tamarix control and related restoration actions are often very slow and small (e.g. González et al. (2017b); Goetz et al. (2022); but see Sher et al. (2018)). In addition, previous meta-analyses in other systems have similarly found that control of a dominant invader does not necessarily improve the condition of the native plant community (Thomas and Reid 2007; Kettenring and Adams 2011).

The negative effect of time on total combined vegetation metrics was likely a function of the short time elapsed between the end of treatment and the start of monitoring. First, due to the disturbance inherent in restoration treatments, indirect outcomes associated with native species are likely to worsen before they can improve (González et al. 2017b). Second, the effect of time is driven, in part, by short-term decreases in undesirable species cover directly following treatment; later, more subtle changes in community composition are, thus, “worse” than the initial major improvement. In addition, many of the studies used a Before-After comparison, meaning treatment may have started at the same time as monitoring; this may obscure underlying mechanisms. Likewise, understorey metrics had a positive relationship with time, likely due to a peak in undesirable pioneer species directly after disturbance followed by the longer-term establishment of more functionally diverse native vegetation (González et al. 2017a). However, it is important to note that the time frame was still relatively short, with an average of 0.8 years between the end of treatment and the end of monitoring. Many changes in the plant community were missed in the absence of long-term monitoring and many of the reported end states of restoration projects may not be indicative of the ecosystem’s broader trajectory; this is a major limitation somewhat inherent to the field, despite frequent recommendations to engage in longer-term monitoring (e.g. Gann et al. (2019)). Indeed, it is possible that, at this timescale, we are only able to observe initial response to treatment itself rather than long-term ecological change following removal of an invader. More recent published studies in this system have incorporated longer-term monitoring (e.g. González et al. (2020a, b); Henry et al. (2023)); we suggest that future work continue to examine long-term ecosystem trajectories following Tamarix control and follow-up on past projects to determine whether there has been sufficient time to see the anticipated effects of restoration efforts.

We did not observe a significant effect of geographical location (in terms of river basin) on any vegetation outcomes other than Tamarix reduction itself. This suggests that, despite differences in environmental conditions (Sher et al. 2020) and dominant treatment strategies (González et al. 2017b) amongst river basins, variation in outcomes is driven by other factors. The greater reduction of Tamarix in the Upper Colorado River Basin may have been caused, in part by greater incidence of the biocontrol beetle, particularly during earlier study years (Bean and Dudley 2018). Given that degree of defoliation (especially over time) and indirect responses to biocontrol have been mixed (González et al. 2017c, 2020b; Sher et al. 2018; Henry et al. 2021), this result is perhaps unsurprising. A lack of large-scale geographic effects on indirect outcomes (e.g. native vegetation, wildlife) may also highlight the need for careful project strategy at a local scale (Shafroth and Briggs 2008); it is likely that small-scale variation within river basins and, crucially, the human decisions around restoration planning (Sher et al. 2020) play a large role in determining whether common restoration goals are met.

We found some evidence to suggest that wildlife may be negatively impacted by Tamarix control in the aggregate, but it was difficult to elucidate trends due to low replication and lack of a comprehensive body of literature across taxa. Though the meta-analysis did not show any significant relationships between fauna and Tamarix control, the vote count found that birds were negatively affected by biocontrol and cut-stump treatments in most reported cases, while herpetofauna were negatively affected by biocontrol in all reported cases. but positively impacted by other treatment methods in most cases. However, this was likely influenced by the low sample sizes, both in terms of outcomes and publications. This synthesis of the literature does show some support for concerns surrounding the effects of Tamarix control on wildlife (e.g. Bateman et al. (2014); Raynor et al. (2017)), but mainly indicates that more research is required before a clear consensus can be reached.

Our results found meta-analysis to be an effective technique for synthesising the literature on control of a well-studied plant invader. In conducting meta-analysis, restrictions on which types of studies can be included have the risk of biasing the results relative to more comprehensive strategies like narrative reviews. In our experience, the specific and stringent requirements for inclusion in meta-analysis (reporting of effect sizes, variance and sample sizes) tend to exclude “grey” literature, older publications and publications that use multivariate modelling techniques for data analysis. Conversely, qualitative tracing and vote counting may offer a greater sample size in terms of publications (nearly half of the publications used for tracing were excluded from meta-analysis), but with a lower strength of evidence due to more assumptions and less granularity. Success rankings involved human interpretation of each publication’s overall “message,” which applied to each publication as a whole rather than in terms of individual comparisons. By this metric, the most common outcomes of papers were “partial success” and “clear success,” respectively. Likewise, as synthesis techniques became more granular, the precision of our findings increased, but it became more difficult to make generalised claims about outcomes; the transition from vote counting “any” effect to statistically significant effects only greatly increased the number of “no change” outcomes and we found few statistically significant effect sizes in the meta-analysis.

Our results do not show evidence for inherent publication bias in meta-analysis; if publications were excluded, based on the meta-analysis requirements in a truly biased manner, we would expect more discrepancy between meta-analysis and other review techniques than was observed. Success rankings, based on language used in publication abstracts, did show some bias in favour of positive outcomes, but this is more likely a result of authors “putting a positive face” on their work than publications with negative outcomes being left out of other analyses; there was not a significant difference in success rankings amongst papers included only in tracing, tracing + vote count or tracing + vote count + meta-analysis. The tendency towards relatively optimistic language in publication abstracts may also be related to issues surrounding a common lack of clearly stated a priori goals and objectives in restoration projects (Bernhardt et al. 2005; Palmer et al. 2005; Brudvig and Catano 2021). In addition, some projects had stated or implicit goals relating solely or largely to reduction in Tamarix abundance, so it was more likely that these would show “clear success” than a project with more varied goals surrounding indirect responses of other ecosystem components. This is consistent with a previous finding that river restoration outcomes are often reported more optimistically than is accurate, partially due to vague goals and objectives (Jähnig et al. 2011). The relative lack of observed publication bias may be due, in part, to the fact that we focused on one study system rather than attempting to synthesise responses to a conceptual hypothesis across systems (see Gurevitch and Hedges (1999); Koricheva and Gurevitch (2014)). We also did not find evidence that the “file drawer effect” (tendency for negative results to remain unpublished) affected our conclusions, based on our calculated fail-safe number.

Many combinations of treatment methods and ecosystem components, including all combinations showing significant negative relationships, had very low replication in terms of both cases and publications. As a result, for many ecosystem component/treatment combinations, our conclusions are essentially the same as the primary sources themselves. A particular artefact of limitations in paper selection for the meta-analysis is that, while several sources have stated that Tamarix control reduces fire risk and severity assuming dead Tamarix is not left in the system (Drus 2013), the only fire metrics used in our analysis came from a single paper that found the opposite (Drus et al. 2013). In the vote count, we saw very different results between metrics reported to have significant change versus any change at all; with the requirement of reported significance, “no change” was often the most common outcome. This was consistent with the high variance we saw in the meta-analysis and speaks to the importance of reporting statistical significance of results. Of the total 1,461 cases used in the vote count, 837 did not have statistical significance reported. In addition, several newer papers conduct multivariate modelling and do not report group means and variances; these also had to be excluded from our meta-analysis. Further, many sources do not report numerical data in a way that allows for meta-analysis; for this reason, our quantitative results do not necessarily reflect the entire body of literature on Tamarix control (especially given that, despite our efforts to include non-journal sources, we were unable to access many government reports through our searches). Many sources did not report sample sizes or applicable measures of variance. Our conclusions regarding issues with monitoring and reporting are consistent with many prior review papers on this topic (e.g. Kettenring and Adams (2011); Wortley et al. (2013); Morandi et al. (2014); Ruiz-Jaen and Aide (2005); González et al. (2015); Dufour et al. (2019)), all of which found that the scope of monitoring is often limited in scale, time and breadth of ecosystem components. The limited scope of monitoring is also linked with underlying issues regarding a lack of explicitly stated goals and objectives of restoration projects. Despite consistent recommendations for clearer goal-setting in riparian restoration (Landers 1997; Bernhardt et al. 2005; Palmer et al. 2005; Shafroth et al. 2008; Mcdonald et al. 2016; Gann et al. 2019), unclear goals remain a common criticism of recent projects (Kroll et al. 2019; Brudvig and Catano 2021).

The issue of lingering uncertainty is by no means limited to our study system; the field of restoration ecology remains young and norms continue to be established regarding monitoring and evaluation, with implementation limited by logistical constraints (Kettenring and Adams 2011; Gann et al. 2019). Regardless, we urge practitioners and scientists working in this field to consider under-studied aspects of the ecosystem, to report data that meet the standards for meta-analysis and to better enable the science of monitoring by defining clear baselines, goals and expectations for projects. Prior work has found that there is successful information exchange between science and practice regarding best approaches to Tamarix control (Clark et al. 2019), indicating a positive trajectory for better understanding of broad-scale outcomes. The philosophical shift away from a single-species approach and towards one that encompasses the entire ecosystem has been an important development in this field and is indicative of overall directions in restoration ecology, but monitoring of restored systems has typically fallen short of addressing whether control measures have contributed to whole-ecosystem success.