The different types of epistemic and linguistic uncertainty emerge across various stages relevant to an ICAT assessment; first, uncertainties will arise when the impact observation is initially observed and/or measured; second, when the impact is communicated in some form of report and third, when the ICAT assessment is conducted (Figure 1). Any uncertainty that arises at any one stage will continue to be present at all subsequent stages, with uncertainty propagating throughout the process, from the initial impact observation to the final ICAT assessment. Thus, all uncertainties that arise prior to the impact assessment are encapsulated in the subsequent stages (Figure 1). All uncertainties relevant here are included in the impact report box of Figure 1.

Figure 1.

Uncertainties propagate across the process of an impact assessment. The first source of uncertainty emerges due to natural variation associated with the occurrence of an alien species’ impact on native biota. Uncertainties arise at three key stages when information on the impact of an alien species is captured 1) the impact observation stage; i.e. when the impact is measured 2) the impact report stage; i.e. when the impact is communicated in some form of report and finally, 3) at the ICAT assessment stage; i.e. when the assessment is conducted. Any uncertainty that arises will be carried through to the subsequent stages, as illustrated through the encapsulation of uncertainties across the process.

Uncertainty initially emerges in the form of natural variation, which corresponds to spatial and temporal changes occurring within the study system. An appropriate study design will identify a suitable temporal and spatial scale under which impacts of the alien species can be characterised (Christie et al. 2019).

The next step at which uncertainties emerge is when the impact is observed and measured. Here, four new sources of epistemic uncertainties are identified: measurement error, systematic error, model uncertainty and subjective judgement (Figure 1). Each of these uncertainties may not necessarily be relevant for every impact report as the ICAT assessments allow the use of different information sources (see Table 3 for the key differences in impact records between EICAT and SEICAT that should be considered). For instance, media reports of a change in local human activities-in response to an alien species-deriving from interviews with residents will not be subject to model uncertainty.

Although currently not directly addressed in the framework guidelines (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018), linguistic uncertainties are important for assessors to consider when informing the confidence score. Linguistic uncertainties are of direct relevance for ICAT assessments: they occur when the impact observations, or measurements, are described in a report with imprecise and inconsistent language. Often linguistic uncertainty will be difficult to reduce retrospectively. In some cases, linguistic uncertainty (such as a vaguely described methodology of the impact study) may mask the ability to identify epistemic uncertainties.

Under the published guidelines, assessors are instructed to capture the key sources of epistemic uncertainty for each impact report and ascribe these to one overall level of confidence (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018). Following the succession of guidelines, the consideration of uncertainty has been somewhat revised. The most recently-revised EICAT guidelines (IUCN 2020a) identify five major sources of uncertainty that the assessor must consider when assigning a confidence score: i) data quality and type ii) spatial and temporal scale and iii) confounding effects iv) study design and v) overall coherence of evidence. These sources of uncertainty are also relevant for SEICAT; however, given that the currency used to measure impact differs between the two frameworks (native species’ populations vs. human activities), interpretation and importance of different uncertainties may vary to fit the criteria and concepts for each framework (Table 3).

When evaluating the magnitude of an impact, the assessor interprets the information contained in the impact report and, when possible, translates this information into one of the five ICAT magnitudes. As impact reports were not aimed at testing the assessment criteria (e.g. which level of organisation of the native population is affected by the alien), the assessor has to interpret the information at hand, a process which inevitably introduces a new source of uncertainty. It may be difficult for ICAT assessors to identify limitations generated by the way the impact was measured and reported. Ideally, authors of an impact study will address limitations with their research; however, ICAT assessors must critically assess all available information (e.g. study design, statistical analyses) to identify potential weakness in the inference of the data. It is at this stage–where the impact measurement is reported—that linguistic uncertainties become relevant and should ideally be recognised by assessors, who should be aware of how language may influence their interpretation of the information.

Assessments will be further compounded by systematic error (i.e. when the assessor systematically decides to include or exclude information that they should otherwise exclude or include) and subjective judgement (Regan et al. 2002). These sources of uncertainty initially become relevant when the assessor conducts a literature review to extract the records of impact for an alien taxon, then decides which fit the framework criteria. For instance, there may be some confusion as to what sources of impact should be included in assessments. Under the EICAT guidelines, impacts are defined as changes to the environment that reduce native biodiversity or alter ecosystem functioning to the detriment of a native species (Hawkins et al. 2015). Therefore, the inclusion of laboratory and mesocosm experiments presents a grey area when considering impact reports. In many cases, such experiments can be informative towards identifying the mechanism(s) through which an alien species impacts on native biodiversity and if native individuals are (potentially) suffering in their performance. However, laboratory and mesocosm studies will always be limited to revealing impacts of MC or MN, given that EICAT measures impacts based on native communities. Therefore, a decline of a natural population or its local extinction cannot be inferred from artificial settings, but such experiments may be useful to provide information about the mechanisms of impact. If assessors include laboratory- or mesocosm-derived sources of information in EICAT assessments, they should be clearly specified as such. Subjective judgement arises due to the interpretation of information; it emerges at the initial impact observation and continues to appear throughout the assessment procedure as each person involved in the process introduces their own form of subjective judgement (Figure 1). An ICAT assessor’s subjective judgement is the primary form of uncertainty that we can minimise by clarifying concepts appropriate to assigning confidence scores and improving the consistency amongst assessors when using the two assessment schemes. Subjective judgement is also relevant to uncertainties when summarising impacts at the species level (see below). Additionally, it must be considered how the written synthesis of ICAT assessments and the justifications of classifications may propagate linguistic uncertainty further.

Uncertainty in impact assessments means that the true impact can be higher or lower than the one assigned. However, assessors may be confident that an impact magnitude is not lower than the one assigned, but could be higher (or vice versa). Thus, uncertainty can be asymmetrically distributed around the assessment value; it may be larger in one direction than in the other. This directionality aspect of uncertainty is currently not captured using the confidence scores, yet may provide important insight to impacts. Using EICAT as an example, it may be that the assessor assigns a minor impact score (MN) to an impact record that robustly demonstrates that an alien taxon affects the performance of individuals of a native species and, thus, is not negligible (i.e. not MC). However, given the study did not address (i.e. measure) whether the impact is causing a decline in the local population, it is not possible to know whether the ‘true’ impact caused by the alien taxon is higher (MO, MR or MV). For instance, studies that assess physiological responses of native species to invasive species do not necessarily relate such effects beyond the individual (i.e. effects on fitness resulting in declining populations) (Graham et al. 2012). Such cases are quite distinct to impact records that sought to quantify population responses to an alien species, yet found no evidence in support of population decline. Since documenting directionality in uncertainty related to each impact record may improve our overall understanding of potential impacts, this information may be particularly useful once several records of impact are obtained for a single species. Directionality in uncertainty, therefore, presents an important facet of uncertainty to recognise when using the ICAT schemes.

Presently, there is no consideration of uncertainty beyond the confidence score assigned to each impact report (IUCN 2020a; Hawkins et al. 2015; Bacher et al. 2018). The ICAT assessment schemes adopt the precautionary principle, whereby the overall classification of an alien taxon is based on the highest magnitude the taxon has reached. Therefore, there is no distinction between species with the same highest impact magnitude, regardless of whether there are few or many accounts of impact. It is also important to acknowledge additional sources of uncertainty which influence the ability to conduct assessments for alien taxa. As these uncertainties occur beyond the individual impact report level, they are not captured by the confidence score as currently described. Uncertainties due to the biases in the collected and the existing (or produced) impact reports contribute to the quality of final assessments, making them of direct relevance when comparing taxa based on ICAT scores. If alien taxa are compared, based on the highest magnitude they have been observed to cause (Hawkins et al. 2015; Bacher et al. 2018), it is pertinent that their highest impact magnitude caused in nature is documented and that these data have been adequately collected and assessed using the ICAT frameworks. It is likely that the more impact reports for an alien species that are produced, collected and assessed, the higher the chance that the maximum impact of the alien taxon will be detected and correctly classified. We recognise three important aspects to evaluate when looking at species-level comparisons: biases in existing data, data collection and data assessment.

Biases in the existing data

The availability of impact records will vary widely within (Evans et al. 2018b) and between taxa (Vilà et al. 2010) and will not necessarily be reflective of impact severity (Evans and Blackburn 2019). Indeed, of the larger taxonomic groups that have been assessed (amphibians, bamboos, birds), the majority of species are classified as data deficient (Evans et al. 2016; Kumschick et al. 2017; Canavan et al. 2019). As biases in biological records (Isaac and Pocock 2015) and within invasion biology are evident (Pyšek et al. 2008), some taxa will be disproportionately represented when conducting literature searches necessary for ICAT assessments. Gaps may be driven by funding availability with regions associated with higher economic status investing more in invasive species research (Pyšek et al. 2008; Bellard and Jeschke 2016). Further, it is usual for a lag time between an alien species becoming established and research effort on the species in the new environment to be observed (Essl et al. 2015; Lyons et al. 2019). Due to this and other reasons, such as the nature and duration of the peer-review process, the dissemination of impacts reports is often delayed (Vilà et al. 2019). Even well-studied species may not have impacts measured that can be easily transferred to ICAT scores, potentially rendering it data deficient or with few reports from which to derive an impact magnitude. For instance, alien species may be well documented to impact via various mechanisms (e.g. predation, competition) under laboratory settings, but poorly represented under natural conditions. Often, biological aspects, related to mechanisms of impact, are well-researched (e.g. dietary overlap, aggressive behaviour) for alien species, but the effects on native biodiversity are not measured, rendering such studies irrelevant to EICAT assessments. Our main suggestion regarding the bias in–or lack of–existing and relevant impact data, is to adapt future impact reports to EICAT criteria: studies should focus more on the changes in the impacted native populations (in natural conditions) and less on the alien populations.

Biases in the data collection

Inconsistencies amongst assessors may be driven from the initial stage of data collection (the literature review), with variation attributed to different search strategies employed by individual assessors (Kumschick et al. 2017). Reproducibility in science is a major topic of discussion (Baker 2016; Fanelli 2018) and how systematic literature searches are conducted is often poorly detailed leading to non-reproducible results (Cooper et al. 2018; Faggion and Diaz 2019). Assessors should be specific on how they conduct their literature searches to promote transparency, which in turn, will facilitate more robust inter-specific comparisons if data requires additional reviewing. Furthermore, documentation of the sources used to score species and the final data for assessments should be published with studies using the assessment schemes (see also Kumschick et al. 2020). Another major difficulty in data accessibility may arise from language barriers that affect the assessor’s ability to collate impact reports. This is likely to be particularly applicable for SEICAT assessments, where it is expected that relevant reports of impacts on human well-being will, more often, be published in local languages. Discussions with people in local languages to identify socio-economic issues arising from the presence of alien species may facilitate assessments of species that are otherwise data deficient and help better understand additional human dimensions of biological invasions. Much regional evidence on the impacts of alien species will be confined to sources of information, such as local government reports and student theses.

Biases in the data assessment

Additional inconsistencies amongst assessors may occur because the criteria of the ICAT frameworks are interpreted and applied differently; individual assessors will inevitably introduce their own level of bias to the process of both assigning impact categories and confidence scores. A recent study by González-Moreno et al. (2019) found variation in scoring species’ impacts amongst assessors for different assessment schemes, including EICAT. Although a level of subjectivity is inevitable, some of this uncertainty may be reduced through improvement in the protocol, such as the refinement of guidelines, which is already reflected in the succession of EICAT guidelines (Blackburn et al. 2014; Hawkins et al. 2015; IUCN 2020a). However, clarification about the changes and ensuring these are effectively communicated will be important to maximise consistency (see Volery et al. 2020, as the application of different versions of the guidelines may further lead to inconsistencies across different assessments. Conducting workshops, training sessions and developing online tools that help guide assessors through the process–giving examples where uncertainty is most likely to arise–might help reduce these uncertainties. Refinements can be made as feedback from assessors identifies more issues that require additional explanation or adaptation.

It is worth noting that, given the variation observed amongst assessors when applying scoring schemes (Matthews et al. 2017; González-Moreno et al. 2019), confidence scores are likely to be subject to a similar level of inconsistency. The accompanying probabilities (Table 1) to each of the three qualitative confidence scores are intended to reduce variation in the interpretation of terms. Indeed, differences in the interpretation of the descriptions of uncertainty are known to occur amongst individuals (Budescu and Wallsten 1985). Presenting linguistic descriptions and corresponding likelihoods can, therefore, reduce the misinterpretation of confidence scoring (Budescu et al. 2014). The degree of consistency amongst assessors when assigning confidence scores should be examined to determine whether refining the expressions of confidence is necessary to reduce potential misinterpretation.