Print
Understanding uncertainty in the Impact Classification for Alien Taxa (ICAT) assessments
expand article infoAnna F. Probert, Lara Volery, Sabrina Kumschick§, Giovanni Vimercati, Sven Bacher
‡ University of Fribourg, Fribourg, Switzerland
§ Stellenbosch University, Matieland, South Africa
Open Access

Abstract

The Environmental Impact Classification for Alien Taxa (EICAT) and the Socio-Economic Impact Classification of Alien Taxa (SEICAT) have been proposed to provide unified methods for classifying alien species according to their magnitude of impacts. EICAT and SEICAT (herein “ICAT” when refered together) were designed to facilitate the comparison between taxa and invasion contexts by using a standardised, semi-quantitative scoring scheme. The ICAT scores are assigned after conducting a literature review to evaluate all impact observations against the protocols’ criteria. EICAT classifies impacts on the native biota of the recipient environments, whereas SEICAT classifies impacts on human activities. A key component of the process is to assign a level of confidence (high, medium or low) to account for uncertainty. Assessors assign confidence scores to each impact record depending on how confident they are that the assigned impact magnitude reflects the true situation. All possible sources of epistemic uncertainty are expected to be captured by one overall confidence score, neglecting linguistic uncertainties that assessors should be aware of. The current way of handling uncertainty is prone to subjectivity and therefore might lead to inconsistencies amongst assessors. This paper identifies the major sources of uncertainty for impacts classified under the ICAT frameworks, where they emerge in the assessment process and how they are likely to be contributing to biases and inconsistency in assessments. In addition, as the current procedures only capture uncertainty at the individual impact report, interspecific comparisons may be limited by various factors, including data availability. Therefore, ranking species, based on impact magnitude under the present systems, does not account for such uncertainty. We identify three types of biases occurring beyond the individual impact report level (and not captured by the confidence score): biases in the existing data, data collection and data assessment. These biases should be recognised when comparing alien species based on their impacts. Clarifying uncertainty concepts relevant to the ICAT frameworks will lead to more consistent impact assessments and more robust intra- and inter-specific comparisons of impact magnitudes.

Keywords

Alien species, confidence score, EICAT, invasive species, risk, SEICAT

Introduction

Understanding the impacts of alien species in their recipient environments is a key research theme in invasion science (Strayer et al. 2006; Pejchar and Mooney 2009; Vilà et al. 2011; Kumschick et al. 2015). However, making comparisons between taxa is difficult as invasions are context-dependent and measurements of impact are not collected using a consistent method (Courchamp et al. 2017). As such, different frameworks have been developed to guide invasion biologists towards more standardised approaches which facilitate comparisons amongst invasion scenarios (Nentwig et al. 2010, 2016; Blackburn et al. 2014). In 2014, Blackburn and colleagues proposed a systematic method for classifying impacts across alien taxa, based on the effects of alien species on native biota. The resulting Environmental Impact Classification System for Alien Taxa (EICAT) (Blackburn et al. 2014; Hawkins et al. 2015) is conceptually based on the International Union for the Conservation of Nature’s (IUCN) Red List of Threatened Species, which uses a ranked classification scheme to determine the global conservation status for individual species (IUCN 2012). Since its publication, the EICAT protocol has been formalised (IUCN 2020a, b; Hawkins et al. 2015) and applied to various groups including birds (Evans et al. 2016, 2018a), amphibians (Kumschick et al. 2017), gastropods (Kesner and Kumschick 2018), some mammals (Hagen and Kumschick 2018), marine fishes (Galanidi et al. 2018) and bamboos (Canavan et al. 2019). More recently, Bacher et al. (2018) proposed an adapted version of the EICAT framework to address socio-economic impacts (SEICAT) caused by alien species. The currency used to measure impact for this scheme is observed changes to human activities and/or well-being and, to date, SEICAT has been applied to amphibians, birds, marine fishes, some mammals and gastropods, in conjunction with the EICAT assessments (Bacher et al. 2018; Evans et al. 2020; Galanidi et al. 2018; Hagen and Kumschick 2018; Kesner and Kumschick 2018).

In the ICAT classification schemes, assessors first conduct a comprehensive literature search to collate all impact records for a given alien species. They then classify each of these impact records into one of the five ICAT semi-quantitative scenarios, according to the magnitude of the impact. For instance, under EICAT, impact magnitudes are hierarchically structured, based on the level of organisation of the native population(s) (i.e. individuals or populations) in which they cause an effect: MC (Minimal Concern; negligible level of impact, but no impact on the performance of native individuals is detected), MN (Minor; the performance (e.g. growth, reproduction) of native individuals is decreased by the alien, but no impact at the native population level is detected), MO (Moderate; the alien causes a decline in at least one native population), MR (Major; the alien causes a local extinction of at least one native population, but this local extinction is reversible, which means that the native species could recolonise the area if the alien population were removed), MV (Massive; the alien causes an irreversible local extinction of at least one native population). If there is no relevant information to derive an impact score, then a species is classified as Data Deficient.

A key aspect of each assessment involves assigning a confidence score for each recorded impact to provide an estimate of uncertainty. Both frameworks adopt a similar approach as the Intergovernmental Panel on Climate Change (IPCC) and the European and Mediterranean Plant Protection Organization (EPPO) to deal with uncertainty (Mastrandrea et al. 2010; Holt et al. 2012; Kenis et al. 2012). The assessor must assign a confidence score of either high, medium or low, based on guiding probabilities (Table 1), to each impact report, depending on how confident they are that the assigned impact magnitude is true i.e. could the actual impact be lower or higher than what is classified. Although several key sources of uncertainty are identified in the guidelines (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018), whether the current consideration of uncertainty is sufficient has not been critically evaluated.

Table 1.

The three current confidence levels (high, medium, low) assigned to individual impact reports using the ICAT frameworks. Guiding probabilities are given in the guidelines to aid the assessor in interpreting their level of confidence into one of the three qualitative categories.

Confidence level Approximate probability of the impact being correct
High ~90%
Medium ~65–75%
Low ~35%

Inadequately accounting for uncertainty when assigning impact magnitudes could lead to incorrect judgement calls and potentially to non-relevant prioritisation and mismanagement of species. Todd and Burgman (1998) demonstrated how incorporating uncertainty into the conservation status of species can cause differences in the assessment outcome, potentially altering conservation priorities. McGeoch et al. (2012) described the uncertainties associated with alien species listing and demonstrated how they produce inconsistencies at the taxonomic and geographic scale. Insufficient handling of uncertainty may not only be detrimental for the native taxa (EICAT) and human societies (SEICAT) that are affected by alien species; it can lead to public distrust in invasion science and reduce the success of future management and restoration programmes (Liu et al. 2011). Failure to effectively capture and communicate uncertainty may lead to ill-informed decisions, causing people to potentially undermine management objectives (Ascher 2004), which is of particular concern to invasive species management where public support is critical for achieving management outcomes (Bremner and Park 2007; Kraus and Duffy 2010; Novoa et al. 2017; Russell and Stanley 2018).

To address potential sources of uncertainty relevant to the ICAT assessments, we evaluate the current consideration when assigning confidence scores, identifying where uncertainties may arise during the assessment process. In the first part of this manuscript, we explain the key concepts and definitions of uncertainty relevant to the ICAT frameworks and map these along the assessment process. We then proceed to identify new sources of uncertainty currently not considered under the framework guidelines and discuss how these may play a role in both the evaluation of information and the final ICAT scores. In doing so, we develop a more comprehensive understanding of uncertainty relevant to ICAT assessments, which may be of conceptual relevance to other aspects of risk assessment, particularly when extracting and evaluating impact information from various sources.

General types of uncertainty and how they can be expressed

Uncertainties arise because our knowledge of systems is incomplete and we often deal with imperfect information; thus, uncertainty is inherent to all scientific research (van der Bles et al. 2019). In some cases, uncertainty can be minimised through the collection of additional information, yet it is impossible to eliminate uncertainty altogether (Regan et al. 2002). In cases where uncertainty cannot be reduced, best practice involves quantification of–and when this is not possible, sufficient acknowledgement of–where uncertainties remain and how they may alter the interpretation of evidence (Fischhoff and Davis 2014). Common expressions of uncertainty in science are usually communicated through quantitative terms such as confidence intervals, standard deviations and probability distributions, but generally, they capture only parts of the overall uncertainty (e.g. measurement error).

A taxonomy of uncertainty applicable to ecological research was described by Regan et al. (2002), who distinguish between two key types of uncertainty: epistemic and linguistic (Table 2). Given their broad applicability to ecological concepts, these expressions of uncertainty are relevant to ICAT assessments and have recently been considered in developing a framework for uncertainty in invasion science (Latombe et al. 2019). Epistemic uncertainties arise because of our limited knowledge of the system of interest. They can generally be reduced with increasing information; however, obtaining a complete understanding of such systems is almost always impractical, hence the necessity to use simplified models to characterise the true state (Regan et al. 2002). Different types of epistemic uncertainty are relevant to the understanding of alien species impacts in general. These include natural variation, measurement error, systematic error, model uncertainty and subjective judgement (Table 2; Regan et al. 2002). Linguistic uncertainties arise because language is imprecise and changes over time cause terminology to be both used inconsistently and open to interpretation (Regan et al. 2002). The different types of linguistic uncertainty include vagueness, context-dependency, ambiguity, indeterminacy of theoretical terms and underspecificity (Table 2). It is clear that linguistic uncertainty has pervaded invasion science, given the numerous attempts to standardise concepts and definitions to improve consistency across the discipline (Wilson et al. 2020; Colautti and MacIsaac 2004; Richardson et al. 2010; Blackburn et al. 2011).

Table 2.

Different types of epistemic and linguistic uncertainties and their definitions which are relevant to the ICAT assessment process (Regan et al. 2002).

Epistemic Linguistic
Natural variation Vagueness
Variations in the variables measured in the study system (e.g. temporally, spatially). Arises since language allows borderline cases. Particularly relevant to ordinal categories (e.g. high, medium, low) where arbitrary and/or poorly defined cut-offs exist.
Measurement error Ambiguity
Imperfections in the measurement equipment or observational techniques which generates random deviation in the measurement data from the true value. Includes operator error and instrument error. When words have more than one meaning and it is unclear which meaning is intended.
Systematic error Context dependence
Bias in the measuring equipment or sampling procedure that generates non-random deviations from the true value (e.g. via poorly-calibrated equipment). This also includes error resulting from the deliberate judgement of a person to exclude (or include) data. Lack of specificity related to the context in which something is to be understood. For example, understanding the meaning of something being “small” requires knowledge as to whether the description refers to an insect or a plant.
Model uncertainty Underspecificity
Arises due to the necessary simplifications (models) used to represent physical and biological systems. Occurs when there is unwanted generality i.e. there is a lack of specificity to ensure complete understanding.
Subjective judgement Indeterminacy of theoretical terms
Occurs as a result of the interpretation of data, often when data are scarce and/or error prone. Particularly relevant to expert judgement. Arises as the meaning of terms can change over time. For instance, this source of uncertainty is particularly relevant to taxonomic terms, which may be subject to revision, leading to changes in the names of species or higher-level groups.

Considering uncertainty for ICAT assessments

Uncertainty directly relevant to the ICAT assessments can be considered at two levels: 1) the impact report level and, 2) the species level. The impact report level is the individual record of impact (of an alien species at a specific location and point in time) that is documented in some form–such as a journal article of grey literature—and assigned an impact score. In contrast, the species level summarises all the individual records of impact for a particular alien taxon (IUCN 2020a).

Uncertainties relevant at the impact report level

The different types of epistemic and linguistic uncertainty emerge across various stages relevant to an ICAT assessment; first, uncertainties will arise when the impact observation is initially observed and/or measured; second, when the impact is communicated in some form of report and third, when the ICAT assessment is conducted (Figure 1). Any uncertainty that arises at any one stage will continue to be present at all subsequent stages, with uncertainty propagating throughout the process, from the initial impact observation to the final ICAT assessment. Thus, all uncertainties that arise prior to the impact assessment are encapsulated in the subsequent stages (Figure 1). All uncertainties relevant here are included in the impact report box of Figure 1.

Figure 1.

Uncertainties propagate across the process of an impact assessment. The first source of uncertainty emerges due to natural variation associated with the occurrence of an alien species’ impact on native biota. Uncertainties arise at three key stages when information on the impact of an alien species is captured 1) the impact observation stage; i.e. when the impact is measured 2) the impact report stage; i.e. when the impact is communicated in some form of report and finally, 3) at the ICAT assessment stage; i.e. when the assessment is conducted. Any uncertainty that arises will be carried through to the subsequent stages, as illustrated through the encapsulation of uncertainties across the process.

Uncertainty initially emerges in the form of natural variation, which corresponds to spatial and temporal changes occurring within the study system. An appropriate study design will identify a suitable temporal and spatial scale under which impacts of the alien species can be characterised (Christie et al. 2019).

The next step at which uncertainties emerge is when the impact is observed and measured. Here, four new sources of epistemic uncertainties are identified: measurement error, systematic error, model uncertainty and subjective judgement (Figure 1). Each of these uncertainties may not necessarily be relevant for every impact report as the ICAT assessments allow the use of different information sources (see Table 3 for the key differences in impact records between EICAT and SEICAT that should be considered). For instance, media reports of a change in local human activities-in response to an alien species-deriving from interviews with residents will not be subject to model uncertainty.

Although currently not directly addressed in the framework guidelines (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018), linguistic uncertainties are important for assessors to consider when informing the confidence score. Linguistic uncertainties are of direct relevance for ICAT assessments: they occur when the impact observations, or measurements, are described in a report with imprecise and inconsistent language. Often linguistic uncertainty will be difficult to reduce retrospectively. In some cases, linguistic uncertainty (such as a vaguely described methodology of the impact study) may mask the ability to identify epistemic uncertainties.

Table 3.

Major sources of uncertainty are identified in the IUCN (2020a) EICAT guidelines. Each source of uncertainty is relevant to both the EICAT and the SEICAT schemes; assessors must consider each source when assigning confidence scores. The two frameworks differ in their currencies used to measure impact (native populations [EICAT] versus human activities [SEICAT]). Therefore, contextual understanding of how these uncertainties may influence confidence scores is required. We highlight some aspects of how considering uncertainty may differ between EICAT and SEICAT below.

Source of uncertainty For EICAT For SEICAT
Presence of confounding effects
Assessors must consider whether invasive species are drivers or passengers of the recorded impact (MacDougall and Turkington 2005; Bellard et al. 2016; Doherty et al. 2016; Blackburn et al. 2019). Further, they must consider whether there is any evidence for additional driver(s) of change causing the observed effects. A major challenge in understanding the impacts of alien species is to disentangle the driving causes of biodiversity declines. Studies/reports range from being simple negative correlations between alien and native populations to before-after-control-impact studies, which may influence the data quality and interpretation (Kumschick et al. 2015; Christie et al. 2019). Often, observed correlations between alien species and native biodiversity loss are reported, but the cause of change is not the alien. For instance, the driving cause of change may be habitat modification that facilitates the alien, which works simultaneously to cause a negative impact on native species. An example would be an alien species that establishes and thrives in an urban area may not be the driver of native bird declines; rather, it could be that loss of resources due to urbanisation are causing the native birds to decline. Alien species altering human activities should be considered in the same way as for EICAT, i.e. the assessor must ask the question “is the alien driving the recorded changes?” However, with SEICAT, given that people can directly communicate the reason for reducing or discontinuing an activity, it may be possible to get a better understanding of the causality behind the recorded impact magnitude with much higher confidence.
Study design
The ICAT frameworks evaluate the different levels of impact, whereby each step change in a category reflects an increase in the order of magnitude of the particular impact so that a new level of organisation is involved (individuals, population, community). A study/report may describe an impact affecting one organisation level (e.g. performance of the individual), but gives no information of relevance to a higher level (e.g. if the impact reduces the population size). This aspect of uncertainty can be captured by considering the directionality of uncertainty for each impact report. A study that is designed to assess the impacts of an alien on the individual performance, but does not capture any information about impacts to the population cannot be assigned higher than an MN. This does not mean that the true impact is not higher and thus, the impact report cannot be assigned a high confidence. High confidence scores can be assigned when the criterion of the magnitude higher than the one assigned has been investigated and found to be not true. Reports relevant for SEICAT may not capture the true level at which the alien is causing an impact. Often, individual people are interviewed to obtain information on the alien’s impacts and their experience may not represent the true state of the entire community.
Data quality and type
Based on the ICAT guidelines (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018), impacts can be classified as either inferred or observed. Assessors might misinterpret the purpose of this distinction, by considering observational studies as the only studies reporting observed impacts (i.e. this presents a form of linguistic uncertainty present in the guidelines). Rather, we assert that assessors should focus only on the quality of the report, given the invasion scenario. Data used to derive EICAT scores are most frequently sourced from primary (i.e. not secondary referencing) and grey literature. A decrease in the size of human activity may not be quantified but inferred from the evidence. For example, studies of diseases and parasites transmitted by aliens affecting humans will rarely report quantitatively on how they affect activities, although the authors may infer such effects. Data used to derive SEICAT scores are more likely to be anecdotal forms of evidence; personal communications and media reports often contain information of relevance to SEICAT. Although anecdotal evidence may be thought of as lower quality information (Bacher et al. 2018), given people can directly communicate behavioural changes in response to alien species, evidence deriving from such information may reveal the true state of impact. However, as SEICAT uses the change in activity size as the measure of impact, information on how many people participate in the activity and on the local population size, is required for high confidence reports.
Spatial scale
Understanding if the impact has been recorded at a relevant spatial scale to capture the assigned impact magnitude accurately. Assessors should ask if the study was conducted on a scale over which native species in the region of interest can be characterised. This requires a basic understanding of what constitutes a local population for a given species. A population can be difficult to delimit given suitable habitat for a species is usually fragmented across a landscape and further, populations are often managed within geopolitical jurisdictions. It may be particularly difficult to discern if an alien taxon causes a decline in population from available data with high confidence. Surveys may make it appear as if the population has declined, when in reality, species that are mobile may avoid areas when an alien species occurs. The ‘focal region’ for SEICAT can be highly variable given densities of human communities. Impacts may be assessed on scales ranging from small villages to large metropolitan areas. Therefore, data about the number of people affected (i.e. those that reduce their activity) and the population size across the geographic scale should be included in the assessment when the information is available.
Temporal scale
In earlier guidelines, the temporal scale at which the impact was previously recorded was not considered important since the ICAT frameworks assign magnitude, based on the highest impact (Hawkins et al. 2015; Bacher et al. 2018). However, it is important to assess how the “true” impact varies with the temporal scale. Uncertainty at the temporal scale is important in two aspects, whereby the study may not capture a relevant time period to detect maximum impact or the study provides an inaccurate snapshot that is not reflective of impact; for instance, the study focused on one season or was just too short to be able to capture any change. Changes in native population size may be limited to only a short period (e.g. seasonally), which generally has little effect on reducing the overall population size. Assessors should consider that the impact report may provide only a snapshot in time and determine how relevant the impact is at a suitable temporal scale. The same issues relating to temporal scale for EICAT are relevant for SEICAT.
Coherence of evidence
At the individual impact report level, assessors must determine whether all the evidence points towards the same direction or whether evidence may be contradictory or ambiguous. A study relevant for EICAT may present conflicting evidence based on different variables measured to determine impact. For instance, a study measuring more than one physiological variable of a native species in response to an alien may indicate both negative and positive effects (e.g. a reduction in height of plant growth but increase in leaf area size). There may be conflicting reports from individuals as to whether an alien species is causing reductions in activity size.

The assessment process

Under the published guidelines, assessors are instructed to capture the key sources of epistemic uncertainty for each impact report and ascribe these to one overall level of confidence (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018). Following the succession of guidelines, the consideration of uncertainty has been somewhat revised. The most recently-revised EICAT guidelines (IUCN 2020a) identify five major sources of uncertainty that the assessor must consider when assigning a confidence score: i) data quality and type ii) spatial and temporal scale and iii) confounding effects iv) study design and v) overall coherence of evidence. These sources of uncertainty are also relevant for SEICAT; however, given that the currency used to measure impact differs between the two frameworks (native species’ populations vs. human activities), interpretation and importance of different uncertainties may vary to fit the criteria and concepts for each framework (Table 3).

When evaluating the magnitude of an impact, the assessor interprets the information contained in the impact report and, when possible, translates this information into one of the five ICAT magnitudes. As impact reports were not aimed at testing the assessment criteria (e.g. which level of organisation of the native population is affected by the alien), the assessor has to interpret the information at hand, a process which inevitably introduces a new source of uncertainty. It may be difficult for ICAT assessors to identify limitations generated by the way the impact was measured and reported. Ideally, authors of an impact study will address limitations with their research; however, ICAT assessors must critically assess all available information (e.g. study design, statistical analyses) to identify potential weakness in the inference of the data. It is at this stage–where the impact measurement is reported—that linguistic uncertainties become relevant and should ideally be recognised by assessors, who should be aware of how language may influence their interpretation of the information.

Assessments will be further compounded by systematic error (i.e. when the assessor systematically decides to include or exclude information that they should otherwise exclude or include) and subjective judgement (Regan et al. 2002). These sources of uncertainty initially become relevant when the assessor conducts a literature review to extract the records of impact for an alien taxon, then decides which fit the framework criteria. For instance, there may be some confusion as to what sources of impact should be included in assessments. Under the EICAT guidelines, impacts are defined as changes to the environment that reduce native biodiversity or alter ecosystem functioning to the detriment of a native species (Hawkins et al. 2015). Therefore, the inclusion of laboratory and mesocosm experiments presents a grey area when considering impact reports. In many cases, such experiments can be informative towards identifying the mechanism(s) through which an alien species impacts on native biodiversity and if native individuals are (potentially) suffering in their performance. However, laboratory and mesocosm studies will always be limited to revealing impacts of MC or MN, given that EICAT measures impacts based on native communities. Therefore, a decline of a natural population or its local extinction cannot be inferred from artificial settings, but such experiments may be useful to provide information about the mechanisms of impact. If assessors include laboratory- or mesocosm-derived sources of information in EICAT assessments, they should be clearly specified as such. Subjective judgement arises due to the interpretation of information; it emerges at the initial impact observation and continues to appear throughout the assessment procedure as each person involved in the process introduces their own form of subjective judgement (Figure 1). An ICAT assessor’s subjective judgement is the primary form of uncertainty that we can minimise by clarifying concepts appropriate to assigning confidence scores and improving the consistency amongst assessors when using the two assessment schemes. Subjective judgement is also relevant to uncertainties when summarising impacts at the species level (see below). Additionally, it must be considered how the written synthesis of ICAT assessments and the justifications of classifications may propagate linguistic uncertainty further.

Directionality of uncertainty

Uncertainty in impact assessments means that the true impact can be higher or lower than the one assigned. However, assessors may be confident that an impact magnitude is not lower than the one assigned, but could be higher (or vice versa). Thus, uncertainty can be asymmetrically distributed around the assessment value; it may be larger in one direction than in the other. This directionality aspect of uncertainty is currently not captured using the confidence scores, yet may provide important insight to impacts. Using EICAT as an example, it may be that the assessor assigns a minor impact score (MN) to an impact record that robustly demonstrates that an alien taxon affects the performance of individuals of a native species and, thus, is not negligible (i.e. not MC). However, given the study did not address (i.e. measure) whether the impact is causing a decline in the local population, it is not possible to know whether the ‘true’ impact caused by the alien taxon is higher (MO, MR or MV). For instance, studies that assess physiological responses of native species to invasive species do not necessarily relate such effects beyond the individual (i.e. effects on fitness resulting in declining populations) (Graham et al. 2012). Such cases are quite distinct to impact records that sought to quantify population responses to an alien species, yet found no evidence in support of population decline. Since documenting directionality in uncertainty related to each impact record may improve our overall understanding of potential impacts, this information may be particularly useful once several records of impact are obtained for a single species. Directionality in uncertainty, therefore, presents an important facet of uncertainty to recognise when using the ICAT schemes.

Uncertainties relevant at the species level

Presently, there is no consideration of uncertainty beyond the confidence score assigned to each impact report (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018). The ICAT assessment schemes adopt the precautionary principle, whereby the overall classification of an alien taxon is based on the highest magnitude the taxon has reached. Therefore, there is no distinction between species with the same highest impact magnitude, regardless of whether there are few or many accounts of impact. It is also important to acknowledge additional sources of uncertainty which influence the ability to conduct assessments for alien taxa. As these uncertainties occur beyond the individual impact report level, they are not captured by the confidence score as currently described. Uncertainties due to the biases in the collected and the existing (or produced) impact reports contribute to the quality of final assessments, making them of direct relevance when comparing taxa based on ICAT scores. If alien taxa are compared, based on the highest magnitude they have been observed to cause (Hawkins et al. 2015; Bacher et al. 2018), it is pertinent that their highest impact magnitude caused in nature is documented and that these data have been adequately collected and assessed using the ICAT frameworks. It is likely that the more impact reports for an alien species that are produced, collected and assessed, the higher the chance that the maximum impact of the alien taxon will be detected and correctly classified. We recognise three important aspects to evaluate when looking at species-level comparisons: biases in existing data, data collection and data assessment.

Biases in the existing data

The availability of impact records will vary widely within (Evans et al. 2018b) and between taxa (Vilà et al. 2010) and will not necessarily be reflective of impact severity (Evans and Blackburn 2019). Indeed, of the larger taxonomic groups that have been assessed (amphibians, bamboos, birds), the majority of species are classified as data deficient (Evans et al. 2016; Kumschick et al. 2017; Canavan et al. 2019). As biases in biological records (Isaac and Pocock 2015) and within invasion biology are evident (Pyšek et al. 2008), some taxa will be disproportionately represented when conducting literature searches necessary for ICAT assessments. Gaps may be driven by funding availability with regions associated with higher economic status investing more in invasive species research (Pyšek et al. 2008; Bellard and Jeschke 2016). Further, it is usual for a lag time between an alien species becoming established and research effort on the species in the new environment to be observed (Essl et al. 2015; Lyons et al. 2019). Due to this and other reasons, such as the nature and duration of the peer-review process, the dissemination of impacts reports is often delayed (Vilà et al. 2019). Even well-studied species may not have impacts measured that can be easily transferred to ICAT scores, potentially rendering it data deficient or with few reports from which to derive an impact magnitude. For instance, alien species may be well documented to impact via various mechanisms (e.g. predation, competition) under laboratory settings, but poorly represented under natural conditions. Often, biological aspects, related to mechanisms of impact, are well-researched (e.g. dietary overlap, aggressive behaviour) for alien species, but the effects on native biodiversity are not measured, rendering such studies irrelevant to EICAT assessments. Our main suggestion regarding the bias in–or lack of–existing and relevant impact data, is to adapt future impact reports to EICAT criteria: studies should focus more on the changes in the impacted native populations (in natural conditions) and less on the alien populations.

Biases in the data collection

Inconsistencies amongst assessors may be driven from the initial stage of data collection (the literature review), with variation attributed to different search strategies employed by individual assessors (Kumschick et al. 2017). Reproducibility in science is a major topic of discussion (Baker 2016; Fanelli 2018) and how systematic literature searches are conducted is often poorly detailed leading to non-reproducible results (Cooper et al. 2018; Faggion and Diaz 2019). Assessors should be specific on how they conduct their literature searches to promote transparency, which in turn, will facilitate more robust inter-specific comparisons if data requires additional reviewing. Furthermore, documentation of the sources used to score species and the final data for assessments should be published with studies using the assessment schemes (see also Kumschick et al. 2020). Another major difficulty in data accessibility may arise from language barriers that affect the assessor’s ability to collate impact reports. This is likely to be particularly applicable for SEICAT assessments, where it is expected that relevant reports of impacts on human well-being will, more often, be published in local languages. Discussions with people in local languages to identify socio-economic issues arising from the presence of alien species may facilitate assessments of species that are otherwise data deficient and help better understand additional human dimensions of biological invasions. Much regional evidence on the impacts of alien species will be confined to sources of information, such as local government reports and student theses.

Biases in the data assessment

Additional inconsistencies amongst assessors may occur because the criteria of the ICAT frameworks are interpreted and applied differently; individual assessors will inevitably introduce their own level of bias to the process of both assigning impact categories and confidence scores. A recent study by González-Moreno et al. (2019) found variation in scoring species’ impacts amongst assessors for different assessment schemes, including EICAT. Although a level of subjectivity is inevitable, some of this uncertainty may be reduced through improvement in the protocol, such as the refinement of guidelines, which is already reflected in the succession of EICAT guidelines (Blackburn et al. 2014; Hawkins et al. 2015; IUCN 2020a). However, clarification about the changes and ensuring these are effectively communicated will be important to maximise consistency (see Volery et al. 2020, as the application of different versions of the guidelines may further lead to inconsistencies across different assessments. Conducting workshops, training sessions and developing online tools that help guide assessors through the process–giving examples where uncertainty is most likely to arise–might help reduce these uncertainties. Refinements can be made as feedback from assessors identifies more issues that require additional explanation or adaptation.

It is worth noting that, given the variation observed amongst assessors when applying scoring schemes (Matthews et al. 2017; González-Moreno et al. 2019), confidence scores are likely to be subject to a similar level of inconsistency. The accompanying probabilities (Table 1) to each of the three qualitative confidence scores are intended to reduce variation in the interpretation of terms. Indeed, differences in the interpretation of the descriptions of uncertainty are known to occur amongst individuals (Budescu and Wallsten 1985). Presenting linguistic descriptions and corresponding likelihoods can, therefore, reduce the misinterpretation of confidence scoring (Budescu et al. 2014). The degree of consistency amongst assessors when assigning confidence scores should be examined to determine whether refining the expressions of confidence is necessary to reduce potential misinterpretation.

Conclusions

To produce robust impact assessments and facilitate the comparison of impacts between taxa, procedures must adequately account for uncertainties (McGeoch et al. 2012). We have highlighted key sources of uncertainty to consider when conducting the ICAT assessments and emphasised the importance of acknowledging all forms of uncertainty even when not directly relevant to informing confidence scores. As uncertainties propagate throughout the various stages of any ICAT assessment (deriving from both the impact measurer/reporter and the ICAT assessor), it is important that they are clearly defined and acknowledged to improve the overall impact assessment procedure. However, it should be noted that it will be impossible to address all types of uncertainty in any framework due to unforeseeable changes in the system under investigation or other unknown unknowns.

As the ICAT frameworks become more readily applied across different taxonomic groups, uncertainties must be appropriately considered to improve the overall ability to correctly classify impacts. By improving the consideration of uncertainty under the ICAT guidelines, we may increase the functionality of the tool for researchers and practitioners. All other things being equal (i.e. control effort, cultural values, positive impacts etc.), species that will be the best candidates for prioritisation will be those that have the highest impact with high corresponding confidence.

Acknowledgements

This paper emerged from a workshop on ‘Frameworks used in Invasion Science’ hosted by the DSI-NRF Centre of Excellence for Invasion Biology in Stellenbosch, South Africa, 11–13 November 2019, that was supported by the National Research Foundation of South Africa and Stellenbosch University. We thank Khensani Nkuna, Susan Canavan, Bianca Hagen and David Kesner for useful discussions on uncertainty. SK acknowledges the support of the DSI-NRF Centre of Excellence for Invasion Biology (CIB) and Stellenbosch University and the South African Department of Forestry, Fisheries and the Environment (DFFtE) nothing that this publication does not necessarily represent the views or opinions of DFFtE or its employees. AFP, LV, GV and SB acknowledge funding from the Swiss National Science Foundation (grant numbers 31003A_179491 and 31BD30_184114) and the Belmont Forum – BiodivERsA International joint call project InvasiBES (PCI2018-092939). We are thankful to Jodie Peyton and Sandro Bertolino for their helpful comments in improving the manuscript.

References

  • Bacher S, Blackburn TM, Essl F, Genovesi P, Heikkilä J, Jeschke JM, Jones G, Keller R, Kenis M, Kueffer C, Martinou AF, Nentwig W, Pergl J, Pyšek P, Rabitsch W, Richardson DM, Roy HE, Saul W-C, Scalera R, Vilà M, Wilson JRU, Kumschick S (2018) Socio-economic impact classification of alien taxa (SEICAT). Methods in Ecology and Evolution 9: 159–168. https://doi.org/10.1111/2041-210X.12844
  • Bellard C, Jeschke JM (2016) A spatial mismatch between invader impacts and research publications. Conservation Biology 30: 230–232. https://doi.org/10.1111/cobi.12611
  • Blackburn TM, Bellard C, Ricciardi A (2019) Alien versus native species as drivers of recent extinctions. Frontiers in Ecology and the Environment 17: 203–207. https://doi.org/10.1002/fee.2020
  • Blackburn TM, Pyšek P, Bacher S, Carlton JT, Duncan RP, Jarošík V, Wilson JRU, Richardson DM (2011) A proposed unified framework for biological invasions. Trends in Ecology & Evolution 26: 333–339. https://doi.org/10.1016/j.tree.2011.03.023
  • Blackburn TM, Essl F, Evans T, Hulme PE, Jeschke JM, Kühn I, Kumschick S, Marková Z, Mrugala A, Nentwig W, Pergl J, Pysek P, Rabitsch W, Ricciardi A, Richardson D, Sendek A, Vilà M, Wilson JRU, Winter M, Genovesi P, Bacher S (2014) A unified classification of alien species based on the magnitude of their environmental impacts. PLoS Biology 12: e1001850. https://doi.org/10.1371/journal.pbio.1001850
  • van der Bles AM, Freeman ALJ, Mitchell J, Galvao AB, Zaval L, Spiegelhalter DJ (2019) Communicating uncertainty about facts, numbers and science. Royal Society Open Science 6: 181870. https://doi.org/10.1098/rsos.181870
  • Budescu DV, Por H-H, Broomell SB, Smithson M (2014) The interpretation of IPCC probabilistic statements around the world. Nature Climate Change 4: 508–512. https://doi.org/10.1038/nclimate2194
  • Canavan S, Kumschick S, Roux JJL, Richardson DM, Wilson JRU (2019) Does origin determine environmental impacts? Not for bamboos. Plants, people, planet 1: 119–128. https://doi.org/10.1002/ppp3.5
  • Christie AP, Amano T, Martin PA, Shackelford GE, Simmons BI, Sutherland WJ (2019) Simple study designs in ecology produce inaccurate estimates of biodiversity responses. Journal of Applied Ecology 56: 2742–2754. https://doi.org/10.1111/1365-2664.13499
  • Cooper C, Booth A, Varley-Campbell J, Britten N, Garside R (2018) Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies. BMC Medical Research Methodology 18: 1–85. https://doi.org/10.1186/s12874-018-0545-3
  • Courchamp F, Fournier A, Bellard C, Bertelsmeier C, Bonnaud E, Jeschke JM, Russell JC (2017) Invasion Biology: Specific Problems and Possible Solutions. Trends in Ecology & Evolution 32: 13–22. https://doi.org/10.1016/j.tree.2016.11.001
  • Doherty TS, Glen AS, Nimmo DG, Ritchie EG, Dickman CR (2016) Invasive predators and global biodiversity loss. Proceedings of the National Academy of Sciences 113: 11261–11265. https://doi.org/10.1073/pnas.1602480113
  • Essl F, Dullinger S, Rabitsch W, Hulme PE, Pyšek P, Wilson JRU, Richardson DM (2015) Historical legacies accumulate to shape future biodiversity in an era of rapid global change. Diversity and Distributions 21: 534–547. https://doi.org/10.1111/ddi.12312
  • Evans T, Blackburn TM, Jeschke JM, Probert AF, Bacher S (2020) Application of the Socio-Economic Impact Classification for Alien Taxa (SEICAT) to a global assessment of alien bird impacts. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 123–142. https://doi.org/10.3897/neobiota.62.51150
  • Evans T, Kumschick S, Blackburn TM (2016) Application of the Environmental Impact Classification for Alien Taxa (EICAT) to a global assessment of alien bird impacts. Diversity and Distributions 22: 919–931. https://doi.org/10.1111/ddi.12464
  • Evans T, Kumschick S, Şekercioğlu ÇH, Blackburn TM (2018a) Identifying the factors that determine the severity and type of alien bird impacts. Diversity and Distributions 24: 800–810. https://doi.org/10.1111/ddi.12721
  • Evans T, Pigot A, Kumschick S, Şekercioğlu ÇH, Blackburn TM (2018b) Determinants of data deficiency in the impacts of alien bird species. Ecography 41: 1401–1410. https://doi.org/10.1111/ecog.03232
  • Fanelli D (2018) Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proceedings of the National Academy of Sciences 115: 2628–2631. https://doi.org/10.1073/pnas.1708272114
  • Galanidi M, Zenetos A, Bacher S (2018) Assessing the socio-economic impacts of priority marine invasive fishes in the Mediterranean with the newly proposed SEICAT methodology. Mediterranean Marine Science 19: 107–123. https://doi.org/10.12681/mms.15940
  • González-Moreno P, Lazzaro L, Vilà M, Preda C, Adriaens T, Bacher S, Brundu G, Copp GH, Essl F, García-Berthou E, Katsanevakis S, Moen TL, Lucy FE, Nentwig W, Roy HE, Srėbalienė G, Talgø V, Vanderhoeven S, Andjelković A, Arbačiauskas K, Auger-Rozenberg M-A, Bae M-J, Bariche M, Boets P, Boieiro M, Borges PA, Canning-Clode J, Cardigos F, Chartosia N, Cottier-Cook EJ, Crocetta F, D’hondt B, Foggi B, Follak S, Gallardo B, Gammelmo Ø, Giakoumi S, Giuliani C, Guillaume F, Jelaska LŠ, Jeschke JM, Jover M, Juárez-Escario A, Kalogirou S, Kočić A, Kytinou E, Laverty C, Lozano V, Maceda-Veiga A, Marchante E, Marchante H, Martinou AF, Meyer S, Minchin D, Montero-Castaño A, Morais MC, Morales-Rodriguez C, Muhthassim N, Nagy ZÁ, Ogris N, Onen H, Pergl J, Puntila R, Rabitsch W, Ramburn TT, Rego C, Reichenbach F, Romeralo C, Saul W-C, Schrader G, Sheehan R, Simonović P, Skolka M, Soares AO, Sundheim L, Tarkan AS, Tomov R, Tricarico E, Tsiamis K, Uludağ A, van Valkenburg J, Verreycken H, Vettraino AM, Vilar L, Wiig Ø, Witzell J, Zanetta A, Kenis M (2019) Consistency of impact assessment protocols for non-native species. NeoBiota 44: 1–25. https://doi.org/10.3897/neobiota.44.31650
  • Graham SP, Freidenfelds NA, McCormick GL, Langkilde T (2012) The impacts of invaders: Basal and acute stress glucocorticoid profiles and immune function in native lizards threatened by invasive ants. General and Comparative Endocrinology 176: 400–408. https://doi.org/10.1016/j.ygcen.2011.12.027
  • Hawkins CL, Bacher S, Essl F, Hulme PE, Jeschke JM, Kühn I, Kumschick S, Nentwig W, Pergl J, Pyšek P, Rabitsch W, Richardson DM, Vilà M, Wilson JRU, Genovesi P, Blackburn TM (2015) Framework and guidelines for implementing the proposed IUCN Environmental Impact Classification for Alien Taxa (EICAT). Diversity and Distributions 21: 1360–1363. https://doi.org/10.1111/ddi.12379
  • Holt J, Leach AW, Knight JD, Griessinger D, MacLeod A, Gaag DJ van der, Schrader G, Mumford JD (2012) Tools for visualizing and integrating pest risk assessment ratings and uncertainties. EPPO Bulletin 42: 35–41. https://doi.org/10.1111/j.1365-2338.2012.02548.x
  • Isaac NJB, Pocock MJO (2015) Bias and information in biological records. Biological Journal of the Linnean Society 115: 522–531. https://doi.org/10.1111/bij.12532
  • IUCN (2020a) Guidelines for using the IUCN Environmental Impact Classification for Alien Taxa (EICAT) Categories and Criteria. Version 1.1, Gland, Switzerland: IUCN.
  • Kenis M, Bacher S, Baker RHA, Branquart E, Brunel S, Holt J, Hulme PE, MacLeod A, Pergl J, Petter F, Pyšek P, Schrader G, Sissons A, Starfinger U, Schaffner U (2012) New protocols to assess the environmental impact of pests in the EPPO decision-support scheme for pest risk analysis*. EPPO Bulletin 42: 21–27. https://doi.org/10.1111/j.1365-2338.2012.02527.x
  • Kesner D, Kumschick S (2018) Gastropods alien to South Africa cause severe environmental harm in their global alien ranges across habitats. Ecology and Evolution 8: 8273–8285. https://doi.org/10.1002/ece3.4385
  • Kumschick S, Bacher S, Bertolino S, Blackburn TM, Evans T, Roy HE, Smith K (2020) Appropriate uses of EICAT protocol, data and classifications. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 193–212. https://doi.org/10.3897/neobiota.62.51574
  • Kumschick S, Measey GJ, Vimercati G, Villiers FA de, Mokhatla MM, Davies SJ, Thorp CJ, Rebelo AD, Blackburn TM, Kraus F (2017) How repeatable is the Environmental Impact Classification of Alien Taxa (EICAT)? Comparing independent global impact assessments of amphibians. Ecology and Evolution 7: 2661–2670. https://doi.org/10.1002/ece3.2877
  • Kumschick S, Gaertner M, Vila M, Essl F, Jeschke JM, Pyšek P, Ricciardi A, Bacher S, Blackburn TM, Dick JTA, Evans T, Hulme PE, Kuhn I, Mrugała A, Pergl J, Rabitsch W, Richardson DM, Sendek A, Winter M (2015) Ecological impacts of alien species: quantification, scope, caveats, and recommendations. BioScience 65: 55–63. https://doi.org/10.1093/biosci/biu193
  • Latombe G, Canavan S, Hirsch H, Hui C, Kumschick S, Nsikani MM, Potgieter LJ, Robinson TB, Saul W-C, Turner SC, Wilson JRU, Yannelli FA, Richardson DM (2019) A four-component classification of uncertainties in biological invasions: implications for management. Ecosphere 10: e02669. https://doi.org/10.1002/ecs2.2669
  • Liu S, Sheppard A, Kriticos D, Cook D (2011) Incorporating uncertainty and social values in managing invasive alien species: a deliberative multi-criteria evaluation approach. Biological Invasions 13: 2323–2337. https://doi.org/10.1007/s10530-011-0045-4
  • Lyons TJ, Tuckett QM, Hill JE (2019) Data quality and quantity for invasive species: A case study of the lionfishes. Fish and Fisheries 20: 748–759. https://doi.org/10.1111/faf.12374
  • MacDougall AS, Turkington R (2005) Are invasive species the drivers or passengers of change in degraded ecosystems? Ecology 86: 42–55. https://doi.org/10.1890/04-0669
  • Mastrandrea MD, Field CB, Stocker TF, Edenhofer O, Ebi KL, Frame DJ, Held H, Kriegler E, Mach KJ, Matschoss PR, Plattner G-K, Yohe GW, Zwiers FW (2010) Guidance Note for Lead Authors of the IPCC Fifth Assessment Report on Consistent Treatment of Uncertainties. Intergovernmental Panel on Climate Change, 7 pp.
  • Matthews J, Velde G van der, Collas FPL, Hoop L de, Koopman KR, Hendriks AJ, Leuven RSEW (2017) Inconsistencies in the risk classification of alien species and implications for risk assessment in the European Union. Ecosphere 8: e01832. https://doi.org/10.1002/ecs2.1832
  • McGeoch MA, Spear D, Kleynhans EJ, Marais E (2012) Uncertainty in invasive alien species listing. Ecological Applications 22: 959–971. https://doi.org/10.1890/11-1252.1
  • Nentwig W, Bacher S, Pyšek P, Vilà M, Kumschick S (2016) The generic impact scoring system (GISS): a standardized tool to quantify the impacts of alien species. Environmental Monitoring and Assessment 188: 1–315. https://doi.org/10.1007/s10661-016-5321-4
  • Novoa A, Dehnen-Schmutz K, Fried J, Vimercati G (2017) Does public awareness increase support for invasive species management? Promising evidence across taxa and landscape types. Biological Invasions 19: 3691–3705. https://doi.org/10.1007/s10530-017-1592-0
  • Pyšek P, Richardson DM, Pergl J, Jarošík V, Sixtová Z, Weber E (2008) Geographical and taxonomic biases in invasion ecology. Trends in Ecology & Evolution 23: 237–244. https://doi.org/10.1016/j.tree.2008.02.002
  • Regan HM, Colyvan M, Burgman MA (2002) A taxonomy and treatment of uncertainty for ecology and conservation biology. Ecological Applications 12: 618–628. https://doi.org/10.2307/3060967
  • Richardson DM, Pyšek P, Carlton JT (2010) A compendium of essential concepts and terminology in invasion ecology. Fifty Years of Invasion Ecology. Wiley-Blackwell, 409–420. https://doi.org/10.1002/9781444329988.ch30
  • Russell JC, Stanley MC (2018) An overview of introduced predator management in inhabited landscapes. Pacific Conservation Biology 24: 371–378. https://doi.org/10.1071/PC18013
  • Vilà M, Gallardo B, Preda C, García-Berthou E, Essl F, Kenis M, Roy HE, González-Moreno P (2019) A review of impact assessment protocols of non-native plants. Biological Invasions 21: 709–723. https://doi.org/10.1007/s10530-018-1872-3
  • Vilà M, Espinar JL, Hejda M, Hulme PE, Jarošík V, Maron JL, Pergl J, Schaffner U, Sun Y, Pyšek P (2011) Ecological impacts of invasive alien plants: a meta-analysis of their effects on species, communities and ecosystems. Ecology Letters 14: 702–708. https://doi.org/10.1111/j.1461-0248.2011.01628.x
  • Vilà M, Basnou C, Pyšek P, Josefsson M, Genovesi P, Gollasch S, Nentwig W, Olenin S, Roques A, Roy D, Hulme PE (2010) How well do we understand the impacts of alien species on ecosystem services? A pan-European, cross-taxa assessment. Frontiers in Ecology and the Environment 8: 135–144. https://doi.org/10.1890/080083
  • Volery L, Blackburn TM, Bertolino S, Evans T, Genovesi P, Kumschick S, Roy HE, Smith KG, Bacher S (2020) Improving the Environmental Impact Classification for Alien Taxa (EICAT): a summary of revisions to the framework and guidelines. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 547–567. https://doi.org/10.3897/neobiota.62.52723
  • Wilson JRU, Datta A, Hirsch H, Keet J-H, Mbobo T, Nkuna KV, Nsikani MM, Pyšek P, Richardson DM, Zengeya TA, Kumschick S (2020) Is invasion science moving towards agreed standards? The influence of selected frameworks. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 569–589. https://doi.org/10.3897/neobiota.62.53243