Eleven commonly used scientifically based protocols developed or applied in Europe for the evaluation of NNS impacts were selected for comparison by consensus in the AlienChallenge COST Action workshop in April 2014 by 36 European experts in NNS risk assessments (Rhodes, Greece) (Table 1). We included all protocols developed and officially used at national or continent level in Europe (e.g. EPPO, Harmonia+ and GB-NNRA) and the main protocols used by European research community (e.g. GISS and FISK). Only the EFSA protocol was discarded from this selection due to the complexity of extracting and processing the data. Furthermore, during the selection we aimed to cover the major types and groups of protocols in order to guarantee enough variability in their characteristics. The selection does not consider risk analysis tools or updates that have become available after 2015, such as AS-ISK (Copp et al. 2016), which replaces FISK and the other -ISK toolkits and complies with the minimum standards NNS risk analysis under Regulation (EU) No 1143/2014 (Roy et al. 2018). Risk assessments are usually divided into four components that consider the potential for a non-native species to enter a region, establish, spread and cause impacts. The selection included impact assessment and risk assessment protocols for which we only compared the sections dealing with spread and impact as they are largely interrelated. Each protocol considers a different method to calculate the final score per species based on the responses (i.e. aggregation method): maximum impact, accumulated impact, categorization matrix or decision trees, an independent summary question, or the combination of any of the previous methods. Owing to the number of protocols used in the present study and their complexity, no attempt was made to standardize variations in score aggregation methods but rather, where possible, to account for this variability during the data analysis as covariates. Some protocols can be applied to any taxon while others are specific to particular groups or habitats (e.g. BINPAS and FISK are used only for aquatic animals, EPPO Prioritization for plants). As such, the number of protocols assessed per species varied depending on the taxonomic group (Table 1). Although all the -ISK toolkits (FISK, FI-ISK, Amph-ISK, MFISK, MI-ISK) were used for their respective taxonomic groups, in the data analyses all the versions were listed under ‘FISK’ because of their high similarity. For the same reason, the EPPO-EIAs for insects/pathogens and plants were listed together.

Each protocol was characterized according to several variables (Table 1): the categories of impact considered (environmental alone or environmental and socio-economic), inclusion of questions on species spread (yes/no), on scoring scale (i.e. three levels, five levels and more than five levels), whether the protocol included a maximum aggregation method (i.e. the largest value of a set of values) to calculate the final score (yes/no), the number of questions requiring input from the assessors and contributing to the final score, and the expertise on the species required to complete the protocol. The latter was based on 63 responses received from an online anonymous questionnaire distributed to all assessors, which included a question asking them to rate their agreement (from 1 = disagree to 5 = fully agree) with the statement: “This protocol requires a high level of expertise on the species”. Assessors answered this question for each protocol after having completed all assessments. The response values were averaged per protocol to provide a single estimate of the level of expertise required for that NNS protocol (Table 1).

A total of 57 species from different taxonomic groups not native to terrestrial, freshwater, and marine environments in Europe were selected (Suppl. material 1: Table S1). Among them, only two species are native to a part of Europe (Arion vulgaris and Dreissena polymorpha). The list of species was elicited by consensus also at the Alien Challenge COST Action workshop in April 2014 (Rhodes, Greece). During the workshops, the experts were grouped according to their taxonomic expertise under the coordination of a taxonomic leader, in order to select a list of species covering a wide range of European climatic regions and habitat types, biological characteristics and the degree and type of impact. While some NNS were widespread, very well studied and with known impacts, some had a localized geographical distribution (Suppl. material 1: Table S1). Each NNS was assigned to a specific taxonomic group and habitat type: terrestrial plants, freshwater plants, terrestrial vertebrates, terrestrial insects, other terrestrial invertebrates, freshwater invertebrates, freshwater fish, marine species, and pathogens. The scientific knowledge available for the NNS was quantified as the number of records in the Web of Science using the accepted scientific name as a query, and biology and ecology research area as filters (retrieved in August 2016). Additionally, the mean and coefficient of variation of the assessor expertise on each species (Suppl. material 1: Table S1) was derived through a self-valuation questionnaire on each assessed NNS using the following classification: 1 = low (the assessor has not worked with the species); 2 = medium (the assessor has not published on the species but has expertise on it through surveys or reports); and 3 = high (the assessor has published on the species).

There is a large variation in methods to implement the different protocols; some are available as downloadable freeware (-ISK toolkits, the ‘NAPRA’ version of the GB-NNRA), as online applications (e.g. Harmonia+, BINPAS), whereas some have to be constructed following the text guidelines (e.g. GISS, EICAT), and others can be obtained as spreadsheets (e.g. GB-NNRA) or databases (e.g. NGEIAAS). To harmonize use of the protocols and facilitate data retrieval, a comprehensive Excel® spreadsheet template was developed to include all the protocols (see Suppl. material 2). The resulting spreadsheet was checked by the authors or owners of each protocol to ensure that it accurately depicted the original protocol whilst matching the common-practice methodology.

Using the protocols selected in the spreadsheet template, 89 assessors independently assessed between three to 11 species (mean = 3.9) of the taxonomic group in their area of expertise (i.e. terrestrial plants, aquatic plants, terrestrial vertebrates, terrestrial insects, other terrestrial invertebrates, freshwater invertebrates, freshwater fish, marine species and pathogens) (Suppl. material 1: Table S1). All assessors were researchers with expertise in biological invasions (PhD or PhD candidate) selected among the participants of the Alien Challenge COST Action by the coordinators of each taxonomic group. The experience of the assessors with NNS impact assessments varied. Most assessors had occasionally participated in NNS risk assessments exercises (59.3%), while 19.7% had never participated and 17.5% had often participated. All NNS were assessed by a minimum of five assessors (maximum eight) (Suppl. material 1: Table S1), yielding a total of 2614 assessments. Before conducting the assessments, the assessors were required to read the impact assessment guidelines provided per protocol and ask questions directly to the protocol developers if needed. When scoring impacts, assessors were instructed to consider Europe as the risk assessment area and the likely worst-case scenario for each NNS. Based on the precautionary principle, protocols recommend scoring the potential impact of NNS based on the available information either from studies for the area of assessment, or from areas with the same invaded habitat in a similar climate. The assessors were instructed to base their assessments on all available literature, information sources and their own expertise, indicating in the assessment the source of the information. The selection of the literature used for the assessment was left at the discretion of the assessor.

Before retrieving the data, each assessment was checked for completeness. Once all NNS assessments were completed, the final scores for each assessment were extracted. To harmonize scores across protocols, all ordinal scores (i.e. protocols with three or five levels as final scoring scale; Table 1) were transformed into numeric values, with the lowest impact as 1 and the maximum as 3 or 5, respectively. Then, all scores were standardized from 0 to 1 using the following equation (S – Smin)/(Smax – Smin), where S represent the score per NNS in each assessment, and Smax and Smin, the maximum and minimum possible scores provided by the protocol (Turbé et al. 2017).

For each NNS and protocol (471 combinations), the mean and the coefficient of variation (CV) of the final score were calculated. The mean was used as the overall score across experts per NNS and protocol, whereas CV was used as an estimate of the consistency of scores across experts, adjusting for the mean value. First, differences in CV among all protocols were tested using a linear mixed model with protocol name as a fixed effect and species nested within taxonomic groups as random effects (i.e. random intercept model). Second, we used multimodel inference (Burnham and Anderson 2002) of linear mixed models to analyze the relationship between the CV and species characteristics (taxonomic group and available knowledge), protocol characteristics (impact categories, spread question included, final scoring scale, whether final scoring was based on maximum score, number of questions and expertise on the species required) and assessor expertise on the species (mean and coefficient of variance). In this set of models, we used the same random effects structure as in the first model but did not include protocol name as a covariate. Model residuals were checked for normality and homoscedasticity and identified the square root as the best transformation for CV. Multi-model inference, based on the all-subsets selection of predictors, was performed using the corrected Akaike’s Information Criterion (AIC_c) keeping the same random effects in all model combinations. For each combination of predictors, Akaike weights (w_i) were calculated. Considering the best models given the selected predictors (ΔAIC_c < 6) (Richards 2008), the relative importance w_+(j) of each predictor j was estimated as the sum of the AIC_c weights across all models in which the selected predictor appeared. Predictors with higher w_+(j) (i.e. closer to 1) have a higher weight of evidence to explain the response variable with the given data. Finally, the average of regression coefficients weighted by w_i within the subset of best models was calculated.

Differences in the mean CV among levels for the categorical variables in the best candidate model (i.e. with the smallest AIC_c) were tested for significance using a Tukey post hoc test. Prior to modelling, continuous predictors for the models above were checked for multicollinearity using Pearson correlations. All variables were selected for further analyses considering the low correlation values found (r < 0.5; Suppl. material 1: Table S2) (Dormann et al. 2013). Continuous variables were centered (deviate from the mean) and scaled (divided by standard deviation) to facilitate interpretation of model coefficients and model convergence (Schielzeth 2010). Finally, in all models explained above we accounted for the variability in the number of assessments per NNS (5 to 8; Suppl. material 1: Table S1) (i.e. sample size effect), including the number of assessments as a covariate (i.e. fixed effect).

Similarities in the scoring of NNS across the different protocols were compared using hierarchical cluster analyses. Cluster analyses of the mean scores per NNS and protocol (calculations described above) were performed using Spearman’s correlation coefficient as a similarity measure and the complete linkage method (i.e. maximum distance between clusters). Using this method, we first carried out a cluster analysis of all NNS across the six protocols common to all taxonomic groups (i.e. GABLIS, GB-NNRA, EICAT, Harmonia+, GISS and NGEIAAS). Then, separate analyses were also performed for four subsets of NNS with common protocols: 1) aquatic and terrestrial plants, 2) aquatic animals (combining freshwater invertebrates, freshwater fish, and marine invertebrates), 3) terrestrial invertebrates (terrestrial insects and other terrestrial invertebrates), and 4) terrestrial vertebrates (Suppl. material 1: Table S1). Pathogens were not included in this analysis due to the low number (n = 3) of species tested. Prior to these analyses in order to account for the variability in the number of assessments per NNS (five to eight; Suppl. material 1: Table S1) (i.e. sample size effect), we calculated the Pearson’s correlation between the mean score per NSS and protocol and the number of assessments performed for all groups of species indicated above. When the correlation was significant for a group of species (p < 0.05) we used simple linear regression models to relate the mean score with the number of assessments per species and used the model’s residuals in subsequent hierarchical analyses. We followed this approach only for plants and aquatic animals based on the significant correlation found (Plants r: -0.17, p < 0.05; Aquatic animals r: -0.17, p < 0.05). Results without this correction were similar, reinforcing the robustness of the results (Suppl. material 1: Fig. S2). All statistical analyses and figures were carried out in R v3.4.1 (R Core Team 2017) using packages lme4, lsmeans, MuMIn and sjPlot to implement and plot mixed models and gplots for the correlation heat maps and dendrograms.

The mean coefficient of variation (CV) of assessor scores per NNS and protocol was 40% (± 37% SD), with 10% (n = 470) showing complete agreement (CV = 0) among assessors but with maximum variability being 223% (four species in ISEIA: Aedes albopictus, Arion vulgaris, Australoheros facetus and Fascioloides magna; two species in EPPO EIA: Diabrotica virgifera and Tuta absoluta). CV was remarkably different among protocols (Fig. 1). ISEIA, EPPO-EIA and Harmonia+ protocols had the highest CV, whereas NGEIAAS and GABLIS protocols showed the lowest values. CV across assessors was better explained by protocol characteristics than by NNS characteristics (Table 2). Scoring scale, expertise required and the use of maximum impact score were the variables with the highest weight of evidence.

Figure 1.

Coefficient of variation (CV) of species scoring across assessors per impact assessment protocol based on linear mixed models controlling for taxonomic group and species as nested random effects and number of assessments per species as fixed effects. Protocols with the same letters above the graph are not significantly different (p < 0.05; Tukey test). Dots indicate the least squares means per protocol. Lines indicate the confidence interval (95%) around the means.

According to Tukey post hoc tests in the best candidate model, protocols using three score levels had significantly lower CV than the protocols using scales with five levels (difference = 0.25, p < 0.001) or more than five levels (difference = 0.29, p < 0.001). However, protocols with five score levels were similar to protocols with more than five levels (p = 0.27). CV across assessors was significantly lower for protocols that required higher expertise than those for which low expertise was required (Table 2). The expertise required per protocol was highly correlated to the overall number of fields in the protocol (i.e. questions, comments, uncertainty and results; Pearson’s r = 0.9) but less with the number of questions actually contributing to the final score calculation (r = 0.5; Suppl. material 1: Table S2). Protocols using the maximum impact score yielded lower CV values. In terms of protocol content, CV was higher when protocols included a NNS spread module but there was no difference depending on the impact types considered (Table 2). The number of questions contributing to final score and impact categories considered did not show significant relations to CV (Table 2). Among NNS and assessor characteristics, only the mean of assessor expertise on each NNS showed a significant negative relationship with CV values (Table 2). Finally, there were some differences in CV among taxonomic groups (Fig. 2). Although not significant, terrestrial vertebrates, terrestrial plants, pathogens and freshwater invertebrates tended to show lower CVs whereas higher values were found for terrestrial insects, other terrestrial invertebrates and freshwater plants. Only terrestrial insects and freshwater plants showed a significantly higher CV than the average across all taxa (Fig. 2).

Figure 2.

Mean regression coefficient and confidence interval (95%) of taxonomic groups (random effects) in the best linear mixed model explaining the coefficient of variation of scores of 57 invasive non-native species for 11 different protocols including all significant species, assessor and protocol characteristics (see Table 2) .

The pair-wise correlations in NNS scores among the six protocols common to all taxa were highly diverse (min–max = 0.16–0.77; mean = 0.55), indicating low consistency in species scores among some protocols (Fig. 3). With respect to taxonomic groups, aquatic animals had the highest mean correlation among protocols, terrestrial invertebrates and plants showed an equally low mean correlation, and terrestrial vertebrates had the lowest correlation levels (Fig. 4). These correlations remained similar when considering only the protocols common to all three taxonomic groups (Suppl. material 1: Fig. S1) and without sample size correction (Suppl. material 1: Fig. S2). Cluster analysis identified two main groups (Fig. 3, Suppl. material 1: Fig. S1): protocols that include only environmental impacts (NGEIAAS, GABLIS, and EICAT) and protocols that include both environmental and socio-economic impacts (GB-NNRA, GISS and Harmonia+). The scorings of Harmonia+ were clearly distinct from the other protocols (indicated by lower correlation values), particularly for plants and terrestrial invertebrates (Figs 3, 4). Similarly, FISK and GABLIS showed relatively low correlation values with the other protocols for aquatic animals and terrestrial vertebrates, respectively (Fig. 4).

Figure 3.

Spearman correlation matrix and hierarchical cluster of species scorings for the protocols common for all species. The color scale indicates the correlation between the species scorings obtained for each protocol pair. In brackets, the mean of all pair-wise correlations.

Figure 4.

Spearman correlation matrix and hierarchical cluster of the species scorings for the protocols common per species group. The color scale indicates the correlation between the species scorings obtained for each protocol pair. In brackets, the mean of all pairwise correlations per group.

Scoring consistency across assessors and for some taxonomic groups was surprisingly low. It is not clear why these large discrepancies occurred even when the assessors were experts in invasion biology within their taxonomic domain. Many factors can influence the interpretations of context dependence found in the scientific literature, which can lead to subjective and inconsistent answers even amongst expert assessors (Gilovich et al. 2002). Heuristics and bias, including intuitive strategies to process information, can lead to variability in expert responses (O’Hagan et al. 2006). For example, experts might score the impact according to the studies with which they feel most familiar (e.g. conducted by colleagues in their region). Similarly, if there is a lack of information on the impacts for a NNS, then the judgement might be biased towards a NNS of the same taxonomic lineage. Alternatively, inconsistencies might be due to inherent uncertainty. For instance, a greater inconsistency for most groups of aquatic taxa may reflect a higher difficulty in determining impacts than for taxa in other environments (Molnar et al. 2008). Finally, these biases could be balanced by anchoring effects where most assessors might assign intermediate levels of impact when there is insufficient information to fulfil the protocol requests.

Part of the variability in consistency was explained by protocol characteristics and the approaches implemented. Protocols with three score levels were more likely to show consistency among assessors than those with five or more levels. However, a three-category scoring system might not be sufficient to discriminate between NNS impacts or magnitude of impacts and rank NNS for prioritisation, because too many species will have the same score. Protocols that select the highest impact among different categories provided higher consistency. By definition, this approach will homogenise the scores towards higher values discarding inconsistencies from less important impacts in a way that results will be more conservative.

Protocols containing questions that required greater expertise on the species yielded higher scoring consistency than simpler protocols. Protocols requiring greater expertise demanded very detailed information about the species (e.g. expected population lifetime in NGEIAAS) that, when available, is very likely to be available only in few studies. Owing to the restricted number of sources of information, the variability in the final score might be low. Complex protocols might be less user-friendly and more time-consuming, but this in itself could increase focus and decrease subjectivity. Exceptions exist, e.g. the -ISK screening (Copp 2013), whereby the protocol is easy to use but the 49 questions require more time to answer than simpler tools such as ISEIA, which has only 12 questions. However, the questions from simple tools such as ISEIA focus mainly on impacts, whereas the -ISK screening tools include a much broader range of questions, such as invasion history, species traits and susceptibility to management measures. The balance between ease of use and time spent is critical as some protocols are meant to be used for the rapid screening of a NNS, whereas others provide more in-depth assessments. For example, NGEIAAS was designed for professional experts who carry out very detailed risk assessments on behalf of government authorities (Gederaas et al. 2012, Sandvik et al. 2013). This issue highlights that although we only selected impact and spread related sections, the present study compares tools intended for different phases of the risk analysis process, i.e. risk identification (e.g. ISEIA, -ISK screening tools), risk assessment (e.g. GB-NNRA, Harmonia+) and impact assessment (e.g. GISS, EICAT). Further studies could look into a detailed comparison across all phases of the risk analysis process in order to highlight those sections that might require improvement.

Regarding assessor and NNS characteristics, the only factor that significantly increased consistency among assessors was their level of expertise with the assessed species. Assessors that had previous experience with the NNS assessed may have had similar high levels of knowledge on that NNS, and this may have led to similar scores. Nevertheless, this situation is infrequent as NNS assessments are more commonly undertaken by persons familiar with the taxonomic group but not necessarily with the NNS being assessed (e.g. NNS not yet present or still rare in the study area). Unexpectedly, consistency was not related to the availability of information about the species (i.e. higher number of WoS records). The simplest explanation is that the number of studies available does not necessarily indicate more studies relevant for impact assessments as the literature on these species could be linked to other research fields in invasion biology not directly associated with their environmental or socioeconomic impacts. It is also relevant to note that different assessors might have had access to different information sources, particularly non-English literature and reports. This might have affected consistency results but we followed standard practices for NNS risk assessments. Further studies could look at these differences providing a base information for the species to be assessed.

The high inconsistency found among assessor’s scores raises high concerns and suggests that assessments conducted by single assessors should be interpreted with caution (Pheloung et al. 1999, Cousens 2008). Expert working group scoring, the use of consensus techniques and reviewing processes can inform the responses of single assessors and therefore reduce uncertainty (Sutherland and Burgman 2015, Vanderhoeven et al. 2017). For NNS lacking information or contrasting data, structured elicitation techniques, such as the Delphi approach, which is based on a feedback and revision process (Mukherjee et al. 2015), can identify and reduce potential sources of bias among experts (Morgan 2014, Sutherland and Burgman 2015). In practice, risk assessments for NNS, in particular those carried out in the plant health sector, are usually done either by groups of experts, as in EPPO pest risk assessment, or using an independent peer reviewer and an editorial-board type vetting procedure, such as in Great Britain (Baker et al. 2008, Booy et al. 2012). The consensus approach is used for plants and plant pests because those assessments are likely to be used in international trade agreements in order to demonstrate robustness (Schrader et al. 2010). However, national or regional impact risk assessments of NNS for blacklists or prioritization purposes are often based on the judgement of a few or single experts. Thus, efforts should be made to involve a panel of experts in the species or the system following elicitation techniques.

Variations among protocols in species scoring are mainly due to the inclusion, or not, of socio-economic impacts. Although socio-economic and environmental impacts are generally correlated (Kumschick et al. 2015a, Vilà et al. 2010), it is almost impossible to predict the magnitude of one impact from the other (Bacher et al. 2018). For example, many NNS, such as agricultural pests and organisms affecting human health, exclusively cause socio-economic impacts (Kenis and Branco 2010, Kumschick et al. 2015b) and, thus, using protocols that include such impacts will affect the impact ranking of NNS under consideration. Furthermore, the perception of socio-economic impacts is likely to vary across stakeholders. Thus, depending on the target audience and objectives of the assessment, different protocols may be used, focusing either on environmental or socio-economic impacts or both together. The majority of the protocols exclusively considered environmental impacts, and there was greater correlation in scores among these protocols. However, the difference between scores was dependent on the taxonomic group under consideration. Ranking of species completely shifted (negative correlation of scores across protocols) when different impact categories were considered for terrestrial vertebrates and plants, but the difference was lower for aquatic animals. This pattern might be due to differences in the relevance of impacts across taxa, with terrestrial vertebrates showing highly contrasting impact types for single species (e.g. high economic impact but low environmental impact) (Vilà et al. 2010). However, differences in scores among taxonomic groups might again also simply reflect differences in the knowledge of their impacts. Impacts of terrestrial vertebrates or plants might be better known than those of aquatic organisms. Testing this hypothesis requires comparing uncertainty scores provided by experts across impact types and taxonomic groups, which could be done with the current dataset in further studies.

Among all protocols, Harmonia+, FISK and GABLIS led to very different scores in comparison to the other protocols. This difference was partly related to the different impact categories considered but also to the inclusion of questions beyond impact (e.g. management in GABLIS and FISK). Finally, the GB-NNRA protocol showed a variable relation with other protocols across taxa: low correlation with protocols only considering environmental impacts for plants and terrestrial invertebrates but high for vertebrates. The final score in the GB-NNRA was not automatically calculated as in the other protocols. Instead, assessors were asked to provide overall summary scores and confidence rankings for the NNS based on the answers provided in previous sections, which include questions that consider both environmental and socio-economic impacts (Baker et al. 2008, Mumford et al. 2010). This approach could have led to the not consistent relation between the GB-NNRA protocol and the others. However, when used as part of the GB risk analysis process (Booy et al. 2012), it aids the NNS risk analysis panel to identify inconsistencies between the assessor’s individual question responses and their overall scores and confidence levels (Mumford et al. 2010).

Several key factors should be taken into account when selecting or designing a NNS risk assessment protocol, such as the aim, the scope, the consistency and the accuracy of the outcomes, and the resources available to perform the assessment (e.g. time or information). As a first step, the suitability of a NNS risk assessment protocol will depend on the scope and aim of the assessment. For instance, if a NNS is already present in the region of interest, assessments on likelihood of entry and establishment are less meaningful than just the assessment of impact. Protocols with different scopes may produce different results in terms of NNS rankings (Lazzaro et al. 2016). As we have shown, even just considering different types of impacts could result in large differences in rankings. Thus, it is crucial not to mix the results from assessment methods that consider different impacts or phases of the invasion process. Furthermore, our results show that even if the focus is only on impact and spread sections, the choice of the protocol is critical because the scoring consistency will depend on the characteristics of the protocol. Three main factors were responsible for these inconsistencies, the choice of the scoring scale, how the final score is summarized and the general expertise required to use the protocol. We recommend using a 5-level scoring, maximum aggregation method and moderate expertise requirements as a good compromise to reduce inconsistency without losing discriminatory power or usability. In general, we also advise protocol developers to perform sensibility tests of consistency before final release or adoption (e.g. Pheloung et al. 1999). This is crucial because if a protocol yields inconsistent outcomes when used by different assessors, then it is likely that decisions taken based on the results could be variable and disproportionate to the actual impacts (Schrader et al. 2012).

Part of the inconsistency might also come from the way the protocol is used in practice (e.g. standardized forms, clear guidelines, selection of assessors, individual vs. group assessments). We propose three main ways to reduce this type of inconsistency. First, irrespectively of the protocol, selecting a group of assessors with high expertise will yield more consistent results. Second, inconsistencies due to linguistic uncertainties (e.g. definitions, formulations, rating) can be reduced by improving the guidelines and with adequate training of the assessors (Vilà et al. 2019). Third, other studies have suggested using expert elicitation methods to reduce inconsistencies (Morgan 2014, Sutherland and Burgman 2015), such as consensus building (Mukherjee et al. 2015) or quality control mechanisms (e.g. peer-review panels). Elicitation methods can reveal whether differences in scoring outcomes between and within protocols reflect true differences in opinion, lack of evidence, or subjective biases due to protocol interpretation (Vanderhoeven et al. 2017). In fact, scientific consensus and robust revisions are crucial for policy implementation (Turbé et al. 2017). Finally, there will always be inconsistencies due to knowledge gaps and subjectivity in the interpretation of the scientific results when there is high context dependency. This might not be a problem in providing a sound evidence-base for decisions on NNS as long as protocols are used transparently and uncertainties are explicitly dealt with through appropriate methods (Vanderhoeven et al. 2017).