To ensure unequivocal use of words pertaining to NNS risk assessment, this paragraph focusses on clarifying and standardizing the terminology used in this essay. In its most general form, ‘risk’ equals the likelihood that harm will occur multiplied by the consequences of that harm. The main standard-setting organizations (e.g. the CAC, regulating food safety; the FAO/WHO, regulating animal health; the IPPC, regulating plant health) consider a ‘risk analysis’ to include several discrete components, categorized as ‘hazard identification’, ‘risk assessment’, ‘risk management’ and ‘risk communication’ (Fig. 1, Fig. 2). Accordingly, the first component of a full NNS risk analysis, the ‘hazard identification’ step, is where it is decided for which species a risk analysis is to be conducted. This can be done proactively, for example when a horizon-scanning exercise has identified a set of potential NNS; or reactively, when an early-detection network has uncovered emerging populations of a NNS. The second component, the ‘risk assessment’, collates scientific evidence pertaining to the species under consideration. Risk assessments take many forms, from experimental manipulations to soliciting expert opinion (Speirs-Bridge et al. 2010; Aven 2017), but are fundamentally tools for obtaining, organizing, presenting and summarizing information for further use in management processes (Fairbrother and Bennett 1999). The risk assessment component forms the cornerstone of risk analysis, and in the context of NNS, it is typically separated into four subcomponents, corresponding to distinct components of invasion, namely evaluation of the potential/likelihood of introduction, establishment, spread and impact (Fig. 1). Commonly used assessment protocols may focus on specific components of risk assessment (e.g. the impact component only, such as the (S)EICAT protocols, Blackburn et al. 2014; Bacher et al. 2018) or may cover all four risk assessment subcomponents (e.g. such as Harmonia+, D’hondt et al. 2015).

Figure 2.

Schematic representation visualizing the main differences between current practices in NNS risk assessment and the framework for evidence mapping proposed here. Building a transparent and searchable database whereby the evidence base is grouped according to the relevance of the geographical area from where the information is reported, the type of publication, study design and impact direction facilitates evaluating the robustness of NNS impact assessments, and makes them more legitimate for policy decisions.

The third component is known as ‘risk management’. Here, decision-makers consider the information and evidence collected in the preceding risk assessment, and weigh it against any other economic, political or societal factors. Risk management thus is about the selection and application of specific management policies, procedures and practices to reduce or mitigate the proliferation of damaging NNS (Abt et al. 2010; Tollington et al. 2017). Fourth and finally, ‘risk communication’, is closely related to the principle of transparency and with the right of societies to participate in the process of decision making. Its major function should be to ensure that all information and opinions essential for effective NNS management are exchanged among stakeholders and incorporated into the decision making process (Goldschmidt 2017).

A main challenge in NNS impact assessment is the ability to evaluate, compare or even predict the magnitude of impacts attributable to a wide range of non-native taxa, often based on limited evidence (Hulme et al. 2013; Blackburn et al. 2014). Progress has been made in devising generic impact scoring protocols that are applicable across a wide range of taxa and habitats. Recent studies have further proposed methods for ensuring quality control and reducing disagreement between expert assessors (Turbé et al. 2017; Vanderhoeven et al. 2017). For example, González-Moreno et al. (2019) propose that in order to reduce inconsistencies in research findings, assessment protocols should use a five-level scoring rule, maximum aggregation of impacts and the moderation of expertise requirements. However, more fundamentally, protocols differ strongly in the kind of information they accept as an evidence base. Firstly, some protocols consider only impacts originating from the (invaded) area (e.g. EICAT) for which the assessment is done whereas others recommend incorporating impact information from other invaded ranges, or even from the native range (e.g. NNRA; Table 1). Secondly, some protocols focus on impacts reported in the peer-reviewed literature only, while others allow any kind of grey literature or expert opinion to be used (Table 2). Thirdly, protocols also may not clearly discriminate between study designs, risking largely anecdotal observations to be considered as equally informative as experimental studies (Table 2). While protocols often include a general level of confidence for the overall (impact) assessment, this typically does not allow one to identify the type or source of uncertainty. Consequently, reasons for self-reported low levels of confidence in the output of (impact) assessments remain mostly unexplained (Vanderhoeven et al. 2017).

Finally, multiple protocols invoke the precautionary principle as a guideline for building and interpreting the impact evidence base (Table 3). Impact assessment protocols typically consist of a list of questions regarding invader impacts for distinct impact categories. Using the precautionary principle as justification, several protocols instruct expert assessors to give a single answer (i.e. a single impact score) for each category, using only the most severe impact case study encountered – thereby effectively ignoring studies showing less severe impacts. Additionally, when aggregating the answers across impact categories into an overall score, multiple protocols also refer to the precautionary principle when recommending to rank NNS based on the most severe impact scores only. This is controversial, as the consensus view is that the precautionary principle is relevant only for the risk ‘management component’, and not in risk ‘assessment’ (Aven 2011; Ahteensuu and Sandin 2012, Fig. 1, Fig. 2, Box 1). Indeed, the precautionary principle is a normative principle that allows policy-makers to opt for certain cost-effective measures when there are threats of serious or irreversible damage, even if there is a lack of full scientific certainty (UNCED 1992, European Commission 2000, Ahteensuu and Sandin 2012, Garnett and Parsons 2017).

These issues underlying current NNS impact assessments are problematic for several reasons. First, even if conducted for the same species and the same region, differences across protocols in the evidence utilized may lead to inconsistencies in NNS rankings (Matthews et al. 2017). This hinders the acceptance and uptake of NNS biosecurity strategies by decision makers and the general public (McGeoch et al. 2012). Second, NNS impact reports typically derive from a wide range of sources. Although observational and experimental peer-reviewed studies are an important source of impact information, many impacts are only reported in the grey literature. Especially when data is scarce, all available information can be valuable – whether it comes from grey or peer-reviewed literature. This, however, makes it especially relevant to explicitly document which data is considered, as for instance, assessments that include anecdotal information versus experimental data only inherently differ in the quality of their underlying evidence. Accepting different quality levels may result in inconsistencies in assessment outcomes (White et al. 2019). A transparent and systematic classification of the evidence base, which allows going back to the sources, is crucial to avoid a ‘data laundering’ process (Strubbe et al. 2011), whereby stakeholders use the results of the impact assessments to draw conclusions without being aware of the potentially limited quality of the underlying evidence. Third, the rise of ‘invasive species denialism’ (Ricciardi and Ryan 2017; Russell and Blackburn 2017) challenges invasion biologists to better present the available evidence, because disagreements often arise when uncertainty on impacts is confounded by differences in personal values. Minimizing social conflict in NNS management will need more than evidence alone. For example, Crowley et al. (2017a) advocate for the implementation of social impact assessments (‘SIA’) for identifying, evaluating and addressing social costs and benefits, and to enable meaningful public participation in management planning. Hence, especially in our contemporary ‘post-truth’ world (Higgins 2016), compiling and presenting a transparent and publicly available evidence base for informed risk assessment, risk management, and risk communication should be a core concern for invasion ecologists.

To address these challenges, we propose a framework by which all available information is systematically classified and catalogued, in order to achieve the creation of a transparent, objective and reproducible evidence base. Multiple impact assessment protocols already require expert evaluators to document the most severe impact case studies encountered (e.g. Blackburn et al. 2014; Bacher et al. 2018), but similar reporting of all literature sources assessed typically is not mandatory. Here, instead of following protocol-specific instructions regarding eligible evidence, we suggest separating out the initial construction of the evidence base of NNS impacts (Fig. 2), making this independent of the scoring protocol. This evidence base can subsequently be used for impact scoring in combination with a chosen impact assessment protocol (and provided to stakeholders, the general public, and/or reviewers). We propose that this evidence base (a) must include each and every source documenting an invader impact – not only the most severe case study, and (b) that each and every source is catalogued using at least the following four variables (see Table 4 for a summary).

A first important yet variable decision NNS assessment protocols make (see Table 1) is deciding from which ‘geographical area’ invader impacts are included. Invasion impacts are context-dependent, as they, among others, depend on the environment in which the impact occurs (Pyšek and Richardson 2010; Bartz and Kowarik 2019). For a flexible implementation of this decision, we propose that invader impacts are classified in the evidence base depending on whether the information comes from (a) the geographical area for which the assessment is performed, (b) from other non-native areas invaded by the species, (c) from the species’ native range, or (d) from captivity or cultivation. We acknowledge that there may be border cases where it could be difficult to allocate a specific study to one category or the other. Such ambiguity could be commented upon in the database so that other assessors can investigate these cases and consider categorizing them differently. Assessors may also consider investigating how alternative allocations affect the final conclusions (i.e. a sensitivity analysis). Choosing which geographical areas are relevant can depend on the goal of the assessment. For example, the EICAT protocol (Blackburn et al. 2014) is based only on impacts that have been observed in the invaded area under consideration. Other protocols explicitly aim at quantifying not only the actual, but also the potential impact invaders can have in the invaded area under consideration, by incorporating impacts recorded elsewhere as a proxy (Bomford 2008; Nentwig et al. 2016, see Table 1). Indeed, both Matthews et al. (2017) and Verbrugge et al. (2010) found that the geographical area considered was a root cause of variability among NNS impact classifications. As an example, depending on the European country where the impact information was taken from, the fish species Umbra pygmaea is classified either as a low priority ‘non-invasive’ introduced species or as a ‘high risk’ invader (Verbrugge et al. 2010). When impacts are clearly labelled based on their geographical context, assessors can transparently debate and decide which evidence is or is not incorporated into the specific assessment they are carrying out, and the consequences of using different criteria on invasive species impact rankings can be transparently assessed and discussed.

Second, evidence should be classified according to its “source type”, as either (a) peer-reviewed literature, (b) non-peer-reviewed (”grey”) literature or (c) unpublished data (i.e. personal communication, personal observation, unpublished data). NNS assessment protocols again differ in which source types are included (Table 2), so an evidence base that is structured accordingly allows for flexible use and evaluation of this criterion. For example, the fact that Chlamydia psittaci (the bacterium causing the zoonotic infectious disease psittacosis) is present in Belgian non-native ring-necked parakeet (Psittacula krameri) populations is only known from a single line in a grey literature report (Vangeluwe et al. 2004). Thus, the type of publication that is allowed into the evidence base, or its weight, may have a marked effect on the assessment outcome and on the identification of which kinds of impact may be most threatening.

Third, the evidence should also be explicit about the ‘study design’. This is important, as also peer-reviewed studies can strongly differ in the amount and quality of the evidence they provide. Therefore, we propose to classify the study design as either (a) an experimental study, i.e. any study using a qualitative/quantitative experimental manipulation of the mechanisms by which the invader is presumed to have an effect, so causality can be inferred, (b) a non-experimental study, i.e. any study using a qualitative/quantitative scientific sampling design to quantify associations between NNS and impacts, without being able to definitively establish causality, (c) an anecdotal report, i.e. any casual observation acquired without a qualitative/quantitative scientific sampling design, so presence/absence of impact can be inferred but neither magnitude nor causality, or (d) indirect reports, i.e. data not observed by the person reporting it, or sources that do not report primary data, so impacts cannot be verified.

Fourth, the evidence should include the direction of the impacts encountered (Schlaepfer et al. 2011; Tanner et al. 2017; Dickey et al. 2018; Hagen and Kumschick 2018). Impact assessments typically only consider deleterious impacts (Baker et al. 2008), and often do not take into account evidence of no impact. Besides deleterious impacts, NNS can also have beneficial impacts, and information on such beneficial impacts may be used by policy-makers in the subsequent risk management and risk communication steps. Indeed, recent European Union legislation aimed at combatting NNS explicitly states that risk assessments should include “a description of the known uses for the species and social and economic benefits deriving from those uses” (EU Regulation 1143/2014, Art. 5, 1(g)). Along similar lines, Branquart et al. (2016) mention that when NNS impacts are offset against perceived gains, cataloguing such gains belongs within the scope of the broader risk analysis. A concrete example of such possible beneficial effects is shown by the invasion by Scotch broom (Cytisus scoparius) in New Zealand, where this plant is considered valuable by beekeepers (Jarvis et al. 2006), whereas farmers and the forestry industry consider it a pest and opt for releasing biocontrol agents. Including direction of impacts therefore will also highlight that impacts (in either direction) may not be fully objective and can be “user-dependent”: some impacts may be scored differently by distinct sections of the scientific community and the general public. We therefore strongly advocate that information on absent and (apparent) beneficial invader impacts is fully included in a transparent and systematic manner in the evidence database. By making evidence of beneficial impacts part of the evidence base, policy-makers or conservation managers can rely on this information in later stages of the risk analysis process, i.e. in the risk management step (Fig. 1, Fig. 2) where any other relevant economic, political or societal factors are considered.

Our framework aims at strengthening the existing standards towards transparency and reproducibility in NNS impact assessments, in line with current scientific trends. For example, recently, Galanidi et al. (2018) applied the EICAT protocol for prioritizing marine invasive fishes in the Mediterranean and published the full underlying evidence base, detailing for each impact encountered (not only the worst ones) the geographical area where the impact was recorded and the study design of the manuscript reporting it. In that sense, our proposal to build and publish the evidence base responds to the needs identified by the scientific community. The criterion ‘geographical area’ refers to the issue of transferability of non-native species impacts, while study design and source type are both associated with the credibility and reproducibility of scientific findings. Direction of impact is included here because of its relevance for the broader risk analysis process and this information is valuable in the later stages following risk management. We aim to promote such organization and reporting of impact evidence. We stress that, here, we do not make a judgement about which evidence should or should not be considered in invasive species impact assessment, but we do call for an evidence base that includes all known information, allowing to transparently track which decisions any study may have made. We further note that our framework would also facilitate the interchange or publication of data sets. This can prevent unnecessary replication of literature review efforts, facilitate rapid updating, enable comparison of outcomes of assessments with respect to different assessment protocols, and promote the involvement of other stakeholders. Technical barriers for sharing data have recently been lowered by the emergence of digital platforms such as, for example, Data Dryad, Figshare, Zenodo, or the GBIF Integrated Publishing Toolkit (Groom et al. 2017). Impact assessment databases can be made findable, shareable and citable using resolvable Digital Object Identifiers (DOI, Kahn and Wilensky 2006).

Fig. 3 provides a hypothetical example illustrating that impact scores can be strongly dependent on whether certain source and evidence types, or geographical areas, are included or excluded in the final impact assessment. Classifying the evidence base according to the variables outlined above prior to the actual scoring allows one to transparently track why impact classifications may differ between scoring protocols. White et al. (2019) recently applied the evidence base scheme proposed here to first collate evidence on impacts caused by parakeets and then leveraged this information to carry out a GISS-based impact assessment for all parakeet species introduced to Europe. They found that the types of evidence included in assessments strongly influenced outcomes, whereby, for example, including evidence from the native range or anecdotal evidence resulted in a switch from minimal-moderate to moderate-major overall impact scores (Fig. 4). Such transparency is important for application of assessment outcomes by different users and supports the communicability and acceptance of assessment results (Bartz and Kowarik 2019). We should however note that uncertainty in NNS impact assessments has many sources (reviewed by McGeoch et al. 2012). Our framework may help by addressing the ‘epistemic’ uncertainty due to data quality. Having all information on available evidence at hand will help assessors assigning impact scores and the associated uncertainty. For example, an assessor could opt for maximum scoring (and assign the invader to a ‘high impact’ category) but report a large degree of uncertainty (because the evidence for the more severe impacts comes from grey literature studies). That is one of the ways we envisage our evidence base will facilitate invasive species impact assessments.

Figure 3.

Hypothetical example of an impact assessment carried out following the evidence mapping framework presented here. The figure shows how impact evidence can differ across dimensions, and that consequently, in- or excluding certain classes of evidence (and how they are scored) can strongly change impact assessment final outcomes. The size of the dots is proportional to the number of impacts reported in the literature. In this hypothetical example, including impact evidence from native-range grey literature would result in the species under consideration being assigned a far higher threat level compared to only considering peer-reviewed studies from the invaded area under consideration. Ranking NNS based on their threat level can be done either by averaging impact scores across impact categories (‘mean impact scoring’), or based solely on the most severe impact recorded (‘worst-case impact scoring’).

Figure 4.

Effect of allowing only impact evidence from the invaded area under assessment (orange lines) versus also including evidence from other geographical areas (brown lines) on impact assessment outcomes for ring-necked (left) and monk parakeets (right) introduced to Europe. Impacts were scored according to the GISS protocol (Nentwig et al. 2016), whereby the magnitude of impact is quantified with six levels ranging from 0 (no impacts known) to 5 (the highest possible impact, see Table 2 in Nentwig et al. 2016). Spider graphs are drawn using maximum scoring (i.e. based on the worst recorded impact for each impact mechanism) based on data from White et al. (2019).

Both according to scholars and to legal entities such as the EU, the precautionary principle is part of the risk ‘management’ step, allowing policy-makers to take certain decisions even if there is no full scientific certainty or agreement (European Commission 2000; Ahteensuu and Sandin 2012, Fig. 1, Box 1). Introducing the precautionary principle during the impact scoring in risk ‘assessment’ comes down to reframing this principle from a rule that guides how we should ‘act or take decisions’, to a principle about what we should ‘believe’ (Harris and Holm 2002). Such use of the precautionary principle may be problematic for two reasons. First, the precautionary principle likely is one of the underlying causes of systematic differences between assessment outcomes, as protocols that invoke this principle score invader impacts solely on the worst recorded impacts and consequently tend to result in higher impact scores compared to protocols that do not rely on the precautionary principle. Such disagreement between different protocols can be problematic and undermine the credibility of assessments (see e.g. Vanderhoeven et al. 2017; Turbé et al. 2017; González-Moreno et al. 2019). Second, instead of providing ‘a scientific and objective evaluation’ (European Commission 2000), basing impact assessment scoring on the precautionary principle ‘may encourage assessors to select information that portrays alien species in the worst possible light’ (Matthews et al. 2017). Dahlstrom et al. (2011) remark that variation regarding the incorporation of the precautionary principle can contribute to disparate outcomes of different risk assessment frameworks. Along similar lines, Heard et al. (2011), for example, note that it had become difficult to quantify how widespread a threat the non-native chytrid fungus Batrachochytrium dendrobatidis really is to the world’s IUCN Red Listed amphibians because – motived by the precautionary principle – assessors routinely list the disease as a contributing factor, mostly without any evidence on whether the disease has actually been detected (a more recent analysis does confirm that the chytridiomycosis panzootic is a leading cause of species extinctions at a global scale, Scheele et al. 2019).

The precautionary principle is an important and explicit part of several impact assessment schemes (Table 3). For instance, some protocols, such as AquaNIS, EFSA and EPPO, make no reference to the precautionary principle, while others explicitly invoke the use of this principle when assessing risks and impacts (e.g. EICAT, GISS, Harmonia+; Table 3 for full details). When referenced in NNS impact assessment, we argue the precautionary principle is typically used (1) to limit the evidence base to only the most severe documented impacts, (2) for using the most severe impact recorded as sole criterion for ranking NNS, and (3) to lower the evidence bar needed to accept an impact. In contrast, we contend that the evidence base accompanying impact assessments should include all relevant studies encountered, not only the most severe ones. Next, impact assessments result in impact scores, and these scores need to be integrated to allow ranking invaders according to their overall impacts. While we reject the precautionary principle as justification for using maximum impacts as the sole acceptable criterion for categorizing invaders, maximum scoring can be useful for identifying NNS which may, depending on location or context, have a single, large impact. For example, in Europe, the ruddy duck (Oxyura jamaicensis) threatens the survival of the endangered native white-headed duck (Oxyura leucocephala) through hybridization (Munoz-Fuentes et al. 2007). This ‘most severe’ impact alone formed the justification for an eradication campaign targeting the species (Henderson 2009). Importantly, our evidence framework allows for easy inclusion of alternative criteria. For example, when an invader is capable of damaging its environment in multiple ways, listing all impacts and averaging them (both within impact categories, and when summarizing impact category scores into an overall impact score) allows for a more representative picture on how likely and how high impacts currently really are, even if there is no single, major impact known (D’hondt et al. 2015; Nentwig et al. 2016). Consider, as a theoretical example, an impact assessment scheme that would employ ten different impact mechanisms for which impact scores need to be derived from the literature. Imagine we have two invasive species: species A attains a ‘moderate’ impact score for only one category, while species B attains a ‘minor’ impact in four categories and a ‘moderate’ impact in another two. Maximum scoring will conclude species A and B pose a similar overall threat, while averaging-based scores will assign species B to a higher threat level compared to A. Having an evidence base that includes all known impact case studies additionally allows one to, for example, provide histograms of impact scores, further clarifying the distribution of NNS impact evidence and severity.

Lastly, we argue that calling upon the precautionary principle encourages impact assessors – most likely unintentionally – to give a greater weight to any evidence suggesting an impact, regardless of the origin, type and quality of underlying evidence (Harris and Holm 2002). Quantifying invader impacts is fraught with difficulties (see above, Courchamp et al. 2017), yet, under the guise of the precautionary principle, invasion biology sometimes drifts into strong inferences that species have a greater impact than is objectively justified by the evidence (see for example Strubbe et al. 2011). This may be motivated by concerns that decision-makers will not act when a NNS suspected of damaging its environment is not unequivocally designated as a high-impact invader. Yet, invoking the precautionary principle in impact assessment risks over-emphasizing likely context-dependent impacts and can mask actual differences in impact between species. This not only leads to disagreements between experts on the magnitude of NNS impacts (Crowley et al. 2017b; Davis and Chew 2017; Russell and Blackburn 2017), but may also fuel public opposition to NNS management, especially for so-called charismatic invaders such as most birds and many mammals (Dana et al. 2013; Estévez et al. 2015). In addition, using the precautionary principle in impact assessments may lead to an inflation in the number of species classified as high-impact invaders, straining and potentially misallocating the resources available for NNS management (Matthews et al. 2017). We therefore argue against the use of the precautionary principle during impact assessment.