Print
Is invasion science moving towards agreed standards? The influence of selected frameworks
expand article infoJohn R. U. Wilson§, Arunava Datta§, Heidi Hirsch§, Jan-Hendrik Keet§, Tumeka Mbobo§, Khensani V. Nkuna§, Mlungele M. Nsikani§, Petr Pyšek|§, David M. Richardson§, Tsungai A. Zengeya#, Sabrina Kumschick§
‡ South African National Biodiversity Institute, Cape Town, South Africa
§ Stellenbosch University, Stellenbosch, South Africa
| Czech Academy of Sciences, Průhonice, Czech Republic
¶ Charles University, Prague, Czech Republic
# University of Pretoria, Pretoria, South Africa
Open Access

Abstract

The need to understand and manage biological invasions has driven the development of frameworks to circumscribe, classify, and elucidate aspects of the phenomenon. But how influential have these frameworks really been? To test this, we evaluated the impact of a pathway classification framework, a framework focussing on the introduction-naturalisation-invasion continuum, and two papers that outline an impact classification framework. We analysed how these framework papers are cited and by whom, conducted a survey to determine why people have cited the frameworks, and explored the degree to which the frameworks are implemented. The four papers outlining these frameworks are amongst the most-cited in their respective journals, are highly regarded in the field, and are already seen as citation classics (although citations are overwhelmingly within the field of invasion science). The number of citations to the frameworks has increased over time, and, while a significant proportion of these are self-citations (20–40%), this rate is decreasing. The frameworks were cited by studies conducted and authored by researchers from across the world. However, relative to a previous citation analysis of invasion science as a whole, the frameworks are particularly used in Europe and South Africa and less so in North America. There is an increasing number of examples of uptake into invasion policy and management (e.g., the pathway classification framework has been adapted and adopted into EU legislation and CBD targets, and the impact classification framework has been adopted by the IUCN). However, we found that few of the citing papers (6–8%) specifically implemented or interrogated the frameworks; roughly half of all citations might be viewed as frivolous (“citation fluff”); there were several clear cases of erroneous citation; and some survey respondents felt that they have not been rigorously tested yet.

Although our analyses suggest that invasion science is moving towards a more systematic and standardised approach to recording invasions and their impacts, it appears that the proposed standards are still not applied consistently. For this to be achieved, we argue that frameworks in invasion science need to be revised or adapted to particular contexts in response to the needs and experiences of users (e.g., so they are relevant to pathologists, plant ecologists, and practitioners), the standards should be easier to apply in practice (e.g., through the development of guidelines for management), and there should be incentives for their usage (e.g., recognition for completing an EICAT assessment).

Keywords

Biological invasions, EICAT, introduction pathways, invasion science, Pathway Classification, Unified Framework

Introduction

The field of invasion science has grown rapidly (Pyšek et al. 2006; Richardson and Pyšek 2008). However, despite major advances on many fronts, there are ongoing debates about how the phenomenon of biological invasions should be circumscribed and classified (Latombe et al. 2019). Such differences in definitions hamper our ability to develop robust generalisations, consistently monitor the phenomenon across different scales, and report on it to multiple stakeholders. To facilitate generalisations, and to improve the link between science, policy, and management, numerous frameworks have been developed in an attempt to unify different concepts and definitions. For these frameworks to allow for generalisations and to have value in decision-making, they need to be applicable across taxonomic groups and environments and be accepted by different end users.

These issues were discussed as part of a workshop on “Frameworks in Invasion Science” in November 2019 (Wilson et al. 2020). As background to this workshop, and to understand the role of frameworks in invasion science generally, this paper explores the degree to which existing frameworks have been accepted and adopted. For this purpose, we selected three of what we consider amongst the most influential recent frameworks in invasion science: the pathway classification framework first outlined by Hulme et al. (2008); the proposed Unified Framework for Biological Invasions describing the introduction-naturalisation-invasion continuum (Blackburn et al. 2011); and the Environmental Impact Classification for Alien Taxa [the rationale was introduced by Blackburn et al. 2014; and guidance as to how to apply it in practice (with slight modification) was provided by Hawkins et al. 2015]. These are hereafter referred to as the “Pathway Classification”, the “Unified Framework”, and “EICAT”, respectively (and where data are presented for all three frameworks, they are presented in this order, with a combined/single figure for the two papers that outline EICAT). This is a biased selection. Many more frameworks have been proposed, some of which are very similar to those selected (Catford et al. 2009; Leung et al. 2012; Wilson et al. 2020), and several others paved the way for the frameworks selected here (Nentwig et al. 2010; Richardson et al. 2000; Williamson and Fitter 1996). However, we selected these frameworks as they capture the phenomenon of invasion in its entirety (i.e., introduction dynamics, establishment, spread, and impact) and they were all explicitly designed to be generalisable across taxa and contexts. They are also amongst the most widespread and widely adopted frameworks, for example, the Pathway Classification has been modified and adopted into EU regulations and by the Convention on Biological Diversity (Scalera et al. 2016), and EICAT was adopted by the IUCN (IUCN 2020). Both the Unified Framework and the Pathway Classification have been proposed for use in international biodiversity standards, and EICAT is under consideration for a future proposal (Groom et al. 2019). Therefore, they arguably represent the frameworks that are closest to being standards in invasion science, and see Box 1 for how they have been adopted policy and management settings in South Africa as an example.

Box 1.

How the frameworks have influenced policy and management in South Africa.

All three frameworks–the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; modified by Hawkins et al. 2015)–have been implemented to different degrees in South Africa. While these frameworks are not formally part of South African legislation, they are incorporated into national reporting on biological invasions and in a recently-developed risk analysis framework (see details below). There is, therefore, an incentive for South African researchers to explicitly use the coding of the frameworks.
Status report on biological invasion in South Africa

South African regulations on biological invasions require that, every three years, a report on the status of biological invasions and the effectiveness of control measures and regulations is produced. The primary aim of the status report is to strengthen the links between basic research, policy, and management by detailing the current status and providing support to decision-makers. The first report was released in October 2018 and it was the first effort globally to report on the status of biological invasion at a national level (van Wilgen and Wilson 2018). The report is based around 20 indicators covering pathways, species, sites, and interventions (Wilson et al. 2018). Of these, six indicators require the direct application of the invasion frameworks, and a further two are related to the frameworks.
Risk analysis framework
The South African regulatory lists (Department of Environmental Affairs 2014a; b) were initially developed through a series of stakeholder engagements and expert panel meetings (Kumschick et al. 2020-b). However, this has been contested in some cases. In response to the need for transparent and repeatable evidence to underpin the list, a risk analysis framework was developed. (Kumschick et al. 2020-c) As with the status report, the framework explicitly tries to align with the proposed frameworks.

Due to the way we selected the three frameworks, our analysis is somewhat circular. For example, the frameworks were selected on the basis that there has been some uptake into policy, so it is unsurprising that we found some uptake by policy-makers. However, we feel it is important to: (i) establish whether these frameworks are used broadly by people interested in invasion science or used just by a subset (e.g., only researchers based in Europe or only people studying marine invasions); (ii) determine whether the frameworks are being used as they were intended or only used to justify working on biological invasions; (iii) to assess how users perceive the frameworks; and (iv) to draw insights on how the field could move forward.

Methods

To evaluate the impact of the frameworks, we conducted an analysis of the citations of the papers, surveyed the authors of citing papers, and explored the extent to which the frameworks have been used in policy and management documents.

Citation analysis

The impact of a research publication is often measured by where it is published and how often it is cited (Biagioli 2016). By aggregating across publications, metrics have been developed to provide a measure of the impact of individual scientists and institutions (Hirsch 2005) that is incorporated into decisions around recruitment, promotions, and research funding (Hicks et al. 2015). While such metrics are simple and transparent, they create perverse incentives. For example, researchers, in an attempt to increase their h-scores, might inappropriately or egregiously promote their own work when reviewing or editing other people’s manuscripts (Biagioli 2016; Zaggl 2017). Nonetheless, and acknowledging that impact as measured by citations is a different concept from research quality (Bornmann and Haunschild 2017), citations are a useful starting point to evaluate impact.

We explored four main aspects. First, we assessed the proportion of self-citations to gauge the degree to which the frameworks were only used by those who constructed them. Second, we evaluated whether the geographic and taxonomic biases apparent in the scientific literature in general (cf. Wilson et al. 2007; Wuestman et al. 2019) and invasion science in particular (Pyšek et al. 2006; Pyšek et al. 2008) were also apparent in the papers citing the frameworks. Our expectation was that the selected frameworks would be used across taxa as they were designed to be generally applicable. For example, an explicit rationale for the development of the Unified Framework was to merge a scheme predominately used by zoologists (Williamson and Fitter 1996) with a scheme used predominately by botanists (Richardson et al. 2000). Third, we wanted to explore whether the citing papers actually implemented the frameworks or simply cited the papers to back up general comments about biological invasions. And finally, we wanted to assess the degree to which the citations were from studies focussing on biological invasions or whether the frameworks had impact beyond their originally-intended field of study.

We downloaded bibliographic information from the ISI Web of Science Core Collection (https://www.webofknowledge.com) on 1 July 2019 for all the publications listed as citing one of the four papers considered here (Blackburn et al. 2014; Blackburn et al. 2011; Hawkins et al. 2015; Hulme et al. 2008), and obtained copies of the citing publications if possible (books and book chapters were omitted if a digital copy could not be readily obtained – 3.2, 1.6, 2.8% of cases for the Pathway Classification, the Unified Framework, and EICAT respectively; Suppl. material 1). We developed an initial protocol to score the articles according to set criteria. Ten of the authors scored 10 papers to look at consistency in scoring (i.e., inter-rater reliability). For most categories, it was found to be consistent, but in a few cases (e.g., the discipline), we found there was some disagreement that could be reduced by refining the protocol. However, when attempting to score papers in terms of the degree of influence the frameworks had on the paper there was substantial disagreement, even after discussion to refine the categories [Fleiss’ Kappa of 0.179 in R package irr (Gamer et al. 2019)]. As a result, the extent of influence of each framework was scored by only one person for consistency (JRUW scored the Pathway Classification and SK scored the Unified Framework and EICAT, after discussing and aligning the scoring categories, see Suppl. material 2: Table S2.1). Most authors did some scoring of the other sections. We then adapted the protocol (see Suppl. material 2.1) and scored each paper accordingly (see Table 1 for details of the data extracted). It took 2–10 minutes to score each paper once it was downloaded.

Table 1.

Summary of the results of the citation analysis of frameworks in invasion science. Where numbers are given, they are for the Pathway Classification framework (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; Hawkins et al. 2015) in that order.

Variable Type Description Expectation Results
Discipline Factor with three levels (invasion, ecology, other) A rough indication of what the topic of the paper is. No specific expectation, but provides an indication of the extent to which the frameworks have been used beyond invasion science. The vast majority of citing papers were directly related to biological invasions (96, 93, 92%), but all of the frameworks were cited by some broader ecological (or evolutionary) papers (4, 6, 7%), and a handful of papers in other disciplines (< 1% in each case, including some in journals with apparently no link to ecology, for example, the American Journal of Roentgenology).
Extent of influence Ordered factor with four levels (general, definition, broad, specific) An interpretation of how the citation is actually used (i.e., the degree to which the paper implements the framework). This provided a response variable for testing other variables against. Papers citing the frameworks should tend to implement specific aspects of the proposed frameworks, although they might also be used to make general points about biological invasions. The level of frivolous citations was surprisingly high ~ 50% of all citations were classed as general (i.e. “citation fluff”), with only 6–8% of citing papers actually implementing the frameworks. This pattern was similar across the frameworks studied here. (see Figure 2, cf. the bar widths from left to right).
Self-citation Factor with two levels (TRUE, FALSE) Whether authors of the original paper were also authors of the citing article. There should be a significant number of self-citations, but this should decline over-time as other people start using the framework. Authors of the framework would be more likely to specifically utilise the framework. Confirmed (Figure 1)
There were a large number of self-citations, (25, 22, 39%), but the relative proportion of self-citation is declining over time (generalised linear model with binomial errors, with year as an explanatory variable and whether a reference was a self-citation as the response using Chi-squared test of the change of variance: p = 0.02; p < 0.01; p = 0.11).
Self-citations were more likely to implement the frameworks in detail than use them as “citation fluff” (analysis based on an ordered factor of the extent of influence versus self-citation: F3,417 = 6.1, p < 0.01; F3,708 = 4.0, p < 0.01; F3,240 = 4.8, p < 0.01).
Realm Factor with six levels (freshwater, marine, terrestrial, other, multiple, NA) The scope of the paper in terms of the environment. For laboratory-based studies, this is based on the taxa used. A core rationale for the development of the frameworks was to make them generalisable across different environments (cf. Ojaveer et al. 2018). As such, the expectation was the frameworks are used consistently across realms. Largely confirmed, but with notable biases (Figure 2).
All three frameworks are cited by studies across different realms and taxa, though most were terrestrial studies and most on animals. However, it is notable that, when implemented in detail, the Pathway Classification was particularly used for comparisons across taxa and realms (i.e. multiple), while the Unified Framework tended to focus on particular realms (freshwater or terrestrial) and particular taxa (animals or plants) rather than for comparisons.
Taxon Factor with five levels (animals, plants, other, multiple, NA) The taxonomic scope of the organisms studied in the paper. A core rationale for the development of the frameworks was to make them generalisable across different taxa. As such, the expectation was the frameworks are used consistently across taxa. Largely confirmed, but with notable biases (Figure 2).
See results for realms above. Studies on animals that cited EICAT tended to have implemented the framework more than other studies and there were no fungal or microbial studies as yet.
Number of citations Integer The number of times the framework is cited in the paper. Papers that cite a framework multiple times would be more influenced by those frameworks. Confirmed, but there is a large amount of variation.
The extent of influence tended to increase with number of citations, but this factor on its own did not explain much of the variance in number of citations (31, 23, 30%); and some papers that specifically implemented the frameworks only cited the frameworks once, while other papers that used the frameworks as “citation fluff” still had numerous citations (maximums of 4, 9, 5). In conclusion, the amount of variance explained was not enough to suggest that the number of times the framework is cited in text is a reliable proxy for the extent of influence of the framework on a paper.

The list of journals that cited each framework was extracted. To determine whether the frameworks had impact beyond their originally-intended field of study, we assigned each citing journal to one of three categories – those that explicitly included biological invasions as a subject area; those that published other aspects of ecology or were more general in scope; and those that did not include ecology as a subject area.

To evaluate geographic biases in the papers citing the frameworks, we used the results of a previous analysis of the geographic pattern of invasion science as a whole (Pyšek et al. 2008) as a point of comparison. We identified the corresponding author of studies that had cited the frameworks and assigned their primary affiliation to a geographic region as per the regions used by Pyšek et al. (2008). We then compared the number of studies in each region relative to the number of studies noted in Pyšek et al. (2008) against the expectation based on the rest of the world. After adjusting for multiple comparisons, regions that tended to have cited one of the frameworks either more often or less often than expected were identified (see Suppl. material 2.5).

Survey of citing authors

Because it was difficult to be sure how the frameworks had influenced publications, we surveyed the corresponding authors of papers that cited any one of the four papers. The survey was conducted under ethical clearance (SU project number: 14445) issued by Stellenbosch University.

The questionnaire (Suppl. material 2.2) was structured to assess how the frameworks are viewed and why they were cited in the authors’ works. We used structured questions that were adopted and modified from a framework that has been applied to survey authors in citation analyses (Case and Higgins 2000; Harwood 2008; Prabha 1983; Shadish et al. 1995). The questionnaire consisted of 35 questions in four sections: eight proximity questions that assess the relationship between the person who cited a publication and any authors of that publication; 21 questions that seek reasons why authors might cite a paper; four semi-structured questions to gauge whether the frameworks are used in research or to implement policy and management strategies; and two questions that provided an opportunity to list any suggestions for or proposed improvements to the frameworks (Suppl. material 2.2). The questions that elicit reasons why authors might cite a paper can be grouped into five broad citation categories – classic citations, negative citations, creative citations, personal influence citations, and supportive citations – and the results were interpreted in the context of these groupings. A cover letter and a link to the questionnaire were emailed to a total of 958 corresponding authors, with a reminder sent to non-responders after one week. The survey ran for three weeks, from 13 March to 6 April 2020.

Influence on policy and management

Policy papers and strategies, unlike journal articles, often do not have a comprehensive list of references, are not indexed by academic databases, and many are published in languages other than English. Therefore, we read a selection of national and international policy documents. These documents included national strategies, status reports, national and international guidelines, and documents published by the Convention on Biological Diversity, International Union for Conservation of Nature, and the European Union. We then qualitatively assessed the degree to which the documents explicitly or implicitly referred to or implemented the frameworks. For this purpose, we only considered documents dated more recently than 2008, i.e., after the Pathway Classification was published.

Results

Citation analysis

The results of the citation analysis are summarised in Table 1. As of 1 July 2019, the Pathway Classification had 436 citations recorded on the ISI Web of Science database, the Unified Framework 729 citations, and the two papers that present and refine EICAT 249 citations. This puts them in the top ten most cited papers in their respective journals amongst articles published in the same year or more recently. The vast majority of these citations are from papers that can be classified as invasion science. In fact, about a third of all papers published in the journal Biological Invasions in 2018 cite the Unified Framework. The numbers of citations are increasing annually, with no indication of any plateaus (Figure 1). The number of self-citations has also increased over time, but their relative proportion has declined. Twelve percent of the papers cited more than one of the frameworks (Suppl. material 2.3).

Figure 1.

The number of times each framework paper was cited since publication until 1 July 2019. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (\Blackburn et al. 2014; Hawkins et al. 2015). Values from 2019 only include a portion of the year and even the number of citations by articles published in 2018 is a slight underestimate as it has also increased in the time since July 2019. Each framework has shown a general increase in citations per year since publication, and a decrease in the proportion of self-citations (Table 1).

Almost half of the citing papers only cited the frameworks to justify general comments about biological invasions. Importantly, however, the citing papers covered a wide range of realms and taxa, and the frameworks were implemented in detail in a similar wide range of studies (Figure 2).

Figure 2.

The extent to which the frameworks have influenced citing papers broken down by (a) environment and (b) taxonomic groups. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; Hawkins et al. 2015). The widths of the bars are proportional to the number of citations. The degree to which the framework was used in the citing paper increases from left to right on each figure [from general, to definition, to broad (application), to specific (application)]. The data are in Suppl. material 1, and the methodology used for scoring in Suppl. material 2.1.

The frameworks were cited by articles published in a wide range of journals (151, 223, and 108 journals, see Suppl. material 2.4). Unsurprisingly, the majority of these journals (70, 68, and 79%) have invasions as one of or their main subject area. Similarly, the majority of articles citing each framework (83, 85, and 87%) were explicitly on invasion science. All three frameworks have a global reach and have been cited by authors from around the world working on invasions in a similar global range of sites (Suppl. material 2.5). However, when compared with the analysis of citation patterns in invasion biology (Pyšek et al. 2008), all the frameworks are more frequently cited by researchers based in Europe or South Africa and less often by those based in North America [49, 44, and 52% of all citations to the respective frameworks were from research led by European based authors vs. 22% of all studies in Pyšek et al. (2008); for South Africa: 9, 16, 18% vs. 2%; for North America: 20, 17, 12% vs. 50%; (the probability from a Chi-squared test was < 0.01 in all these cases)]. See Suppl. material 2.5 for the full details.

Survey of citing authors

We received responses from 84 people contacted (~ 9% of the 905 e-mails that did not bounce) from 20 countries (including 14 responses from North America, a slight over-representation). Responses were split fairly evenly across the frameworks (20 of 256; 51 of 589, and 13 of 113 respectively). The statement that respondents tended to most agree with was “This reference is authored by recognized authorities in the field” followed by “This is a classic reference in the field”. The most common reason for citing the frameworks was that they are “classic citations” (Fig. 3, Shadish et al. 1995). Importantly, of the six statements that suggest a paper is viewed as a “classic citation”, the two questions that were not widely supported (in fact more respondents disagreed than agreed) were “There have been substantial efforts to show that the framework is wrong” and “The framework has withstood many efforts to show that it is wrong”. Therefore, while the papers are undoubtedly viewed and used as classic citations, there is a general feeling that the frameworks have not been adequately investigated. This was borne out by various suggestions of how the frameworks could (and in some cases have) been modified or where other frameworks are more appropriate (Suppl. material 2.6).

Figure 3.

Reasons for citing the frameworks based on the response to a questionnaire sent to corresponding authors. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; Hawkins et al. 2015). The categories Negative, Supportive, Personal Influence, Creative, and Classic are based on Shadish et al. (1995). See Suppl. material 2.2 for a copy of the questionnaire and Suppl. material 2.6 for the full results and how the questions map on to different categories.

In terms of the link between the citing authors and the authors of the frameworks, over half have spoken to one of the authors (64, 51, 95%) and a substantial number of these consider one of the authors a personal friend (35, 13, 41%). While the respondents often recommended the citation to others during review (40, 27, 46%), it was not suggested to them often (5, 6, 0%). Of course, the respondents are a small section of the invasion science community who have actively cited the framework and who were willing to respond to a survey concerning the framework.

Influence on policy and management

All three frameworks seem to have had some impact on policy and management (Suppl. material 2.6 and 2.7; Box 1). The Pathway Classification framework has arguably had the most impact. The CBD’s Aichi Biodiversity Target 9 specifies (amongst other things) the need to identify and prioritise pathways by 2020 and a modification of the pathway framework was proposed for use by the CBD itself (Scalera et al. 2016). Other examples of its adoption include the guidelines for invasive species planning and management on islands published by IUCN and the European Union Regulation on the prevention and management of the introduction and spread of invasive alien species. More recently, EICAT has been adopted as an IUCN standard (IUCN 2020) and is anticipated to play an important role in future biodiversity targets and as part of an indicator to track impact (Essl et al. 2020; GEO BON 2015; Latombe et al. 2017). By contrast, we found little evidence that the Unified Framework (the most cited framework investigated here) has been used in policy and management. This could in part be due to differences in how the phenomenon of biological invasions is defined (Wilson et al. 2016). Policy-orientated definitions of invasive organisms often include the impact of the organism, while the biogeographic definition also represented in the Unified Framework (Richardson et al. 2000) seems to be more common in science.

Importantly, once a scientific framework has been widely accepted by an international body like the CBD or the IUCN, it is very likely that the original references are no longer cited. Therefore, caution must be observed in interpreting the policy influence of scientific documents.

Discussion

We found that the invasion frameworks assessed here are widely cited by studies focussing on different realms and taxa, and from many different parts of the world. While many citations might be viewed as frivolous (“citation fluff”), there is a substantial number where the frameworks have been implemented in detail. There are different possible explanations for these trends. Invasion science might be coalescing temporarily; it might be settling down to adopt standard and widely-agreed practices; a particular ‘school’ of invasion science that uses particular frameworks might be emerging; or there might always be a suite of papers that are core papers for citing, but that do not actually influence the direction of the field. We discuss some of these issues here.

Frameworks are temporary, concepts are permanent, but where ideas come from can have long-lasting effects

The Unified Framework and EICAT owe substantial intellectual debts to previous papers and frameworks. Indeed, some of the original frameworks are arguably still more influential. The Unified Framework is based partly on a framework for plants outlined by Richardson et al. (2000), and this earlier paper still tends to be more widely cited. Richardson et al. (2000) had a huge effect on the study of biological invasions. By creating standards that were widely adopted by the research community, data on biological invasions have been increasingly based on a common set of criteria, and are therefore directly comparable. This has facilitated a wide range of comparative analyses [e.g., the Global Naturalized Alien Flora (GloNAF) project (Pyšek et al. 2017)].

Frameworks also evolve and develop over time and in some cases are superseded. The Pathway Classification has been expanded and subcategories developed as part of its proposed uptake by the CBD (Harrower et al. 2017; Scalera et al. 2016). It is noticeable that some more recent journal articles and policy documents implement the CBD pathway classification scheme without citing the original paper on which it is based. This might be quite typical, i.e., once a framework is adopted into a policy or adapted into a guideline, there is a step-change in the impact it has, but conversely, the original paper might no longer be cited. Papers applying EICAT often use it in combination with the Generic Impact Scoring System (GISS; Nentwig et al. 2016; Nentwig et al. 2010) upon which EICAT is based or they use a modification using aspects of both schemes. GISS has probably been more often applied to date and to a wider taxonomic range than EICAT (e.g., Kumschick et al. 2015), but due to EICAT’s adoption as an IUCN Standard, EICAT is rapidly gaining momentum.

Frameworks often need to be adapted in light of practical experience. For example, several adaptations to the Unified Framework have been proposed based on experiences of implementing it in Europe (Groom et al. 2019), Hawaii (Brock and Daehler 2020), and South Africa (Wilson et al. 2018). Similarly, Pergl et al. (2020) and Faulkner et al. (2020) provide proposals to refine the CBD pathway classification scheme based on applying it in different contexts, and Volery et al. (2020) document changes made to EICAT after stakeholder consultation. All three frameworks have, to different extents, been incorporated in developing biodiversity data standards, and this will provide a more formalised process for revising them.

Importantly, however, our results show that the extent of influence of the frameworks is still somewhat affected by how they were originally developed. There is, unsurprisingly, a high level of self-citations, and this likely explains part of the apparent European and South African bias in uptake (cf. Fig. 1 and Suppl. material 2.5). The Pathway Classification was a direct product of the European Union Funded ALARM project (Settele et al. 2005); the idea to develop the Unified Framework arose at a workshop in Switzerland and was further elaborated at a meeting in South Africa; and EICAT resulted from a workshop in Germany. Moreover, of the 33 original authors of the frameworks, 26 are based in Europe, three in South Africa, and one each in Australia, Canada, New Zealand, and the USA. In this context, the global influence of the frameworks has been impressive, but it will be important for the utility of the frameworks to be assessed in more depth and in different contexts. For example, introduction pathways have changed over time (Faulkner et al. 2016; Hulme 2009), and the importance of different pathways varies across the world. Faulkner et al. (2020) highlight one such case, where the trade in traditional medicines is a potentially important introduction pathway in Africa, but is not considered explicitly in the current (arguably Eurocentric) Pathway Classification. Similarly, the Unified Framework originated from combing zoological and botanical frameworks, and there are several practical issues applying both it and the Pathway Classification to fungi and microbes (Paap et al. 2020).

Nonetheless we believe that our results provide some indication that invasion science is beginning to coalesce around systematic schema for classification and understanding that are applicable across taxa and realms.

Comparisons with other frameworks

Given the lack of points of comparison, it is difficult to gauge whether the results seen here are surprising or not. Ideally, we would have looked at the uptake of other important frameworks in invasion science (including historical and more contemporary schemes) and compared with highly-cited framework papers from related disciplines. Unfortunately, the methodology we developed was time-consuming. We found no reasonable proxy for a manual analysis of the extent of influence of the frameworks on the citing papers. There was a broad correlation between our manual scoring of the extent of influence and the number of times a framework was cited, but there were many exceptions (Table 1). Similarly, simply noting whether a citation was in the methods, discussion, or introduction provided some indication of whether the frameworks were used, but not enough to reliably predict that the frameworks were actually implemented (results not shown, but data presented in Suppl. material 1). We concluded that conducting such a citation analysis requires careful examination of at least the sentences that include the citation, and often an evaluation of the whole manuscript. Machine-learning techniques might offer a solution to this issue in future.

We did, however, identify some comparisons that would be particularly interesting and some important research gaps. As mentioned previously, most of the frameworks presented here had progenitors [for example, the Unified Framework explicitly built upon Williamson and Fitter (1996) and Richardson et al. (2000)], an explicit evaluation of how these have been used over time would provide a benchmark against which our results could be assessed. Similarly, while the frameworks chosen reflect pathways, species, and impacts, it would be important to consider frameworks centred around sites of invasion or the effectiveness of interventions (McGeoch et al. 2016; Wilson et al. 2018), or to consider how invasion hypotheses are cited (Catford et al. 2009; Jeschke and Heger 2018). Finally, it might be instructive to track recent frameworks [e.g., SEICAT, the socio-economic impact classification of alien taxa scheme (Bacher et al. 2018), although there has not been much time for uptake].

Insights into citation practices

Our research did not primarily set out to evaluate citation practices, but several insights were apparent. Many of the citations were what we considered frivolous (and informally dubbed “citation fluff”). The introduction of most papers starts with a generic catch-all statement about invasions, and the frameworks were often used to support these, often inappropriately [e.g., citing the Unified Framework as evidence that invasions have impact, or EICAT as a risk assessment protocol (Kumschick et al. 2020-a)]. Arguably “citation fluff” provides an indication of influence, i.e., the frameworks are not directly and explicitly used, but play a role in shaping the overall mental model of the processes at play. However, there were many errors in the way in which the frameworks are cited (see Suppl. material 2.3). Should there be a greater onus on authors, reviewers, and editors to purge “citation fluff” or at least to ensure such references really support the general statements made?

The fact that 20–40% of all citations are self-citations is not necessarily indicative of nepotistic or insular research practices (Seeber et al. 2019). The frameworks were the products of highly productive scientists with the intention of producing seminal papers in a research field in which they were amongst the research leaders. The rapid uptake and declining proportion of self-citations are arguably, healthy signs, as is the geographic spread of the citations. This is borne out by the respondents to the survey where the authors were considered well respected, and the framework papers were, by and large, considered classic papers in the field.

However, the papers analysed and the people surveyed were very biased. The results are, therefore, consistent with the notion of a distinct school of thought amongst certain (particularly European and South African) invasion scientists for whom these frameworks are valuable (cf. the MAFIA framework of Pyšek et al. 2020). A study of researchers who did not cite or use these frameworks despite the framework being relevant to (or even designed to assist) their research would do much to further our understanding of the limitations of the frameworks. As an analogue, it is difficult to understand why some invasions are successful if we lack data on failed invasions (Zenni and Nuñez 2013).

A suggestion to journals – avoid numbered citations

Finally, as a side note, in our experience papers with numeric citations are harder to read, comment on as editors and reviewers, and make analyses, like the one here, much more cumbersome. It is not clear to us why online-only publishers (e.g. the Public Library of Science) persist with this format (cf. https://svpow.com/2011/01/07/an-open-letter-to-plos-one-a-pox-on-your-numbered-references/).

Conclusion

The selected frameworks are influential and widely cited. They are being used to provide information about explicit efforts at monitoring and reporting biological invasions and the development of internationally-agreed data standards. Nonetheless, they are not yet widely implemented as they were originally formulated. We believe that our ability to understand and manage biological invasions will improve as we move increasingly towards agreed standards in the field (Wilson et al. 2020). Invasion frameworks will need to both provide information about such change and be flexible, so they can be modified in the light of the experience and needs of users.

Acknowledgements

This paper emerged from a workshop on ‘Frameworks used in Invasion Science’ hosted by the DSI-NRF Centre of Excellence for Invasion Biology in Stellenbosch, South Africa, 11–13 November 2019, that was supported by the National Research Foundation of South Africa and Stellenbosch University. Workshop participants are thanked for their insight on an earlier draft, Chris Delport and Tamara Robinson for insight into how to score papers, and Suzaan Kritizinger-Klopper for useful discussions. The South African Department of Forestry, Fisheries and the Environment, (DFFtE) are also thanked for funding noting that this publication does not necessarily represent the views or opinions of DFFtE or its employees. PP was supported by EXPRO grant no. 19-28807X (Czech Science Foundation) and long-term research development project RVO 67985939 (Czech Academy of Sciences). DMR acknowledges support from the Oppenheimer Memorial Trust (grant 18576/03). TAZ was supported by the DSI-NRF Centre of Excellence for Invasion Biology (CIB) and National Research Foundation (Grant 103602).

References

  • Bacher S, Blackburn TM, Essl F, Genovesi P, Heikkilä J, Jeschke JM, Jones G, Keller R, Kenis M, Kueffer C, Martinou AF, Nentwig W, Pergl J, Pyšek P, Rabitsch W, Richardson DM, Roy HE, Saul W-C, Scalera R, Vilà M, Wilson JRU, Kumschick S (2018) Socio-economic impact classification of alien taxa (SEICAT). Methods in Ecology and Evolution 9: 159–168. https://doi.org/10.1111/2041-210X.12844
  • Blackburn TM, Essl F, Evans T, Hulme PE, Jeschke JM, Kühn I, Kumschick S, Marková Z, Mrugała A, Nentwig W, Pergl J, Pyšek P, Rabitsch W, Ricciardi A, Richardson DM, Sendek A, Vilá M, Wilson JRU, Winter M, Genovesi P, Bacher S (2014) A unified classification of alien species based on the magnitude of their environmental impacts. Plos Biology 12: e1001850. https://doi.org/10.1371/journal.pbio.1001850
  • Blackburn TM, Pyšek P, Bacher S, Carlton JT, Duncan RP, Jarošík V, Wilson JRU, Richardson DM (2011) A proposed unified framework for biological invasions. Trends in Ecology & Evolution 26: 333–339. https://doi.org/10.1016/j.tree.2011.03.023
  • Bornmann L, Haunschild R (2017) Does evaluative scientometrics lose its main focus on scientific quality by the new orientation towards societal impact? Scientometrics 110: 937–943. https://doi.org/10.1007/s11192-016-2200-2
  • Brock KC, Daehler CC (2020) Applying an invasion and risk framework to track non-native island floras: a case study of challenges and solutions in Hawai‘i. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 55–79. https://doi.org/10.3897/neobiota.62.52764
  • Department of Environmental Affairs (2014a) National Environmental Management: Biodiversity Act 2004 (Act No. 10 of 2004) Alien and Invasive Species Lists, 2014. Government Gazette of South Africa, Pretoria, 3–80 pp.
  • Department of Environmental Affairs (2014b) National Environmental Management: Biodiversity Act 2004 (Act No. 10 of 2004) Alien and Invasive Species Regulations, 2014. Government Gazette, Pretoria, 3–32 pp.
  • Essl F, Latombe G, Lenzner B, Pagad S, Seebens H, Smith K, Wilson JRU, Genovesi P (2020) The Convention on Biological Diversity (CBD)’s Post-2020 target on invasive alien species – what should it include and how should it be monitored? In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 99–121. https://doi.org/10.3897/neobiota.62.53972
  • Faulkner KT, Hulme PE, Pagad S, Wilson JRU, Robertson MP (2020) Classifying the introduction pathways of alien species: are we moving in the right direction? In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 143–159. https://doi.org/10.3897/neobiota.62.53543
  • Faulkner KT, Robertson MP, Rouget M, Wilson JRU (2016) Understanding and managing the introduction pathways of alien taxa: South Africa as a case study. Biological Invasions 18: 73–87. https://doi.org/10.1007/s10530-015-0990-4
  • GEO BON (2015) An essential biodiversity variable approach to monitoring biological invasions: Guide for countries. Group On Earth Observations Biodiversity Observation Network Technical Series #2, 13 pp.
  • Groom Q, Desmet P, Reyserhove L, Adriaens T, Oldoni D, Vanderhoeven S, Baskauf SJ, Chapman A, McGeoch M, Walls R, Wieczorek J, Wilson JRU, Zermoglio PFF, Simpson A (2019) Improving Darwin Core for research and management of alien species. Biodiversity Information Science and Standards 3: e38084. https://doi.org/10.3897/biss.3.38084
  • Harrower CA, Scalera R, Pagad S, Schönrogge K, Roy HE (2017) Guidance for interpretation of CBD categories on introduction pathways. Technical note prepared by IUCN for the European Commission., 100 pp.
  • Harwood N (2008) Citers’ use of citees’ names: Findings from a qualitative interview-based study. Journal of the American Society for Information Science and Technology 59: 1007–1011. https://doi.org/10.1002/asi.20789
  • Hawkins CL, Bacher S, Essl F, Hulme PE, Jeschke JM, Kühn I, Kumschick S, Nentwig W, Pergl J, Pyšek P, Rabitsch W, Richardson DM, Vilà M, Wilson JRU, Genovesi P, Blackburn TM (2015) Framework and guidelines for implementing the proposed IUCN Environmental Impact Classification for Alien Taxa (EICAT). Diversity and Distributions 21: 1360–1363. https://doi.org/10.1111/ddi.12379
  • Hicks D, Wouters P, Waltman L, Rijcke Sd, Rafols I (2015) Nature. The Leiden Manifesto for research metrics 520: 429–431. https://doi.org/10.1038/520429a
  • Hirsch JE (2005) An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America 102: 16569–16572. https://doi.org/10.1073/pnas.0507655102
  • Hulme PE, Bacher S, Kenis M, Klotz S, Kuhn I, Minchin D, Nentwig W, Olenin S, Panov V, Pergl J, Pysek P, Roques A, Sol D, Solarz W, Vila M (2008) Grasping at the routes of biological invasions: a framework for integrating pathways into policy. Journal of Applied Ecology 45: 403–414. https://doi.org/10.1111/j.1365-2664.2007.01442.x
  • IUCN (2020) IUCN EICAT Categories and Criteria. The Environmental Impact Classification for Alien Taxa (EICAT) First edition. IUCN, Gland, Switzerland and Cambridge, 22 pp. https://doi.org/10.2305/IUCN.CH.2020.05.en
  • Kumschick S, Bacher S, Bertolino S, Blackburn TM, Evans T, Roy HE, Smith K (2020) Appropriate uses of EICAT protocol, data and classifications. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 193–212. https://doi.org/10.3897/neobiota.62.51574
  • Kumschick S, Bacher S, Evans T, Markova Z, Pergl J, Pysek P, Vaes-Petignat S, van der Veer G, Vila M, Nentwig W (2015) Comparing impacts of alien plants and animals in Europe using a standard scoring system. Journal of Applied Ecology 52: 552–561. https://doi.org/10.1111/1365-2664.12427
  • Kumschick S, Foxcroft LC, Wilson JR (2020-b) Analysing the risks posed by biological invasions to South Africa. In: van Wilgen BW, Measey J, Richardson DM, Wilson JR, Zengeya TA (Eds) Biological invasions in South Africa. Springer, Berlin, 573–595. https://doi.org/10.1007/978-3-030-32394-3_20
  • Kumschick S, Wilson JRU, Foxcroft LC (2020) A framework to support alien species regulation: the Risk Analysis for Alien Taxa (RAAT). In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 213–239. https://doi.org/10.3897/neobiota.62.51031
  • Latombe G, Canavan S, Hirsch H, Hui C, Kumschick S, Nsikani MM, Potgieter LJ, Robinson TB, Saul W-C, Turner SC, Wilson JRU, Yannelli FA, Richardson DM (2019) A four-component classification of uncertainties in biological invasions: implications for management. Ecosphere 10: e02669. https://doi.org/10.1002/ecs2.2669
  • Latombe G, Pyšek P, Jeschke JM, Blackburn TM, Bacher S, Capinha C, Costello MJ, Fernández M, Gregory RD, Hobern D, Hui C, Jetz W, Kumschick S, McGrannachan C, Pergl J, Roy HE, Scalera R, Squires ZE, Wilson JRU, Winter M, Genovesi P, McGeoch MA (2017) A vision for global monitoring of biological invasions Biological Conservation 213: 295–308. https://doi.org/10.1016/j.biocon.2016.06.013
  • Leung B, Roura-Pascual N, Bacher S, Heikkilä J, Brotons L, Burgman MA, Dehnen-Schmutz K, Essl F, Hulme PE, Richardson DM, Sol D, Vilà M (2012) TEASIng apart alien species risk assessments: a framework for best practices. Ecology Letters 15: 1475–1493. https://doi.org/10.1111/ele.12003
  • McGeoch MA, Genovesi P, Bellingham PJ, Costello MJ, McGrannachan C, Sheppard A (2016) Prioritizing species, pathways, and sites to achieve conservation targets for biological invasion. Biological Invasions 18: 299–314. https://doi.org/10.1007/s10530-015-1013-1
  • Nentwig W, Bacher S, Pysek P, Vila M, Kumschick S (2016) The generic impact scoring system (GISS): a standardized tool to quantify the impacts of alien species. Environmental Monitoring and Assessment 188: 13. https://doi.org/10.1007/s10661-016-5321-4
  • Ojaveer H, Galil BS, Carlton JT, Alleway H, Goulletquer P, Lehtiniemi M, Marchini A, Miller W, Occhipinti-Ambrogi A, Peharda M, Ruiz GM, Williams SL, Zaiko A (2018) Historical baselines in marine bioinvasions: Implications for policy and management. PLoS ONE 13: 48. https://doi.org/10.1371/journal.pone.0202383
  • Paap T, de Beer ZW, Migliorini D, Nel WJ, Wingfield MJ (2018) The polyphagous shot hole borer (PSHB) and its fungal symbiont Fusarium euwallaceae: a new invasion in South Africa. Australasian Plant Pathology 47: 231–237. https://doi.org/10.1007/s13313-018-0545-0
  • Paap T, Wingfield MJ, Burgess TI, Hulbert JM, Santini A (2020) Harmonising the fields of invasion science and forest pathology. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 301–332. https://doi.org/10.3897/neobiota.62.52991
  • Pergl J, Brundu G, Harrower CA, Cardoso AC, Genovesi P, Katsanevakis S, Lozano V, Perglová I, Rabitsch W, Richards G, Roques A, Rorke SL, Scalera R, Schönrogge K, Stewart A, Tricarico E, Tsiamis K, Vannini A, Vilà M, Zenetos A, Roy HE (2020) Applying the Convention on Biological Diversity Pathway Classification to alien species in Europe. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 333–363. https://doi.org/10.3897/neobiota.62.53796
  • Prabha CG (1983) Some aspects of citation behavior – a pilot-study in business administration. Journal of the American Society for Information Science 34: 202–206. https://doi.org/10.1002/asi.4630340305
  • Pyšek P, Bacher S, Kühn I, Novoa A, Catford JA, Hulme PE, Pergl J, Richardson DM, Wilson JRU, Blackburn TM (2020) MAcroecological Framework for Invasive Aliens (MAFIA): disentangling large-scale context dependence in biological invasions. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 407–462. https://doi.org/10.3897/neobiota.62.52787
  • Pyšek P, Pergl J, Essl F, Lenzner B, Dawson W, Kreft H, Weigelt P, Winter M, Kartesz J, Nishino M, Antonova LA, Barcelona JF, Cabezas FJ, Cardenas D, Cardenas-Toro J, Castano N, Chacon E, Chatelain C, Dullinger S, Ebel AL, Figueiredo E, Fuentes N, Genovesi P, Groom QJ, Henderson L, Inderjit, Kupriyanov A, Masciadri S, Maurel N, Meerman J, Morozova O, Moser D, Nickrent D, Nowak PM, Pagad S, Patzelt A, Pelser PB, Seebens H, Shu WS, Thomas J, Velayos M, Weber E, Wieringa JJ, Baptiste MP, van Kleunen M (2017) Naturalized alien flora of the world: species diversity, taxonomic and phylogenetic patterns, geographic distribution and global hotspots of plant invasion. Preslia 89: 203–274. https://doi.org/10.23855/preslia.2017.203
  • Pyšek P, Richardson DM, Jarošík V (2006) Who cites who in the invasion zoo: insights from an analysis of the most highly cited papers in invasion ecology. Preslia 78: 437–468.
  • Pyšek P, Richardson DM, Pergl J, Jarošík V, Sixtová Z, Weber E (2008) Geographical and taxonomic biases in invasion ecology. Trends in Ecology & Evolution 23: 237–244. https://doi.org/10.1016/j.tree.2008.02.002
  • Scalera R, Genovesi P, Booy O, Essl F, Jeschke J, Hulme P, McGeoch M, Pagad S, Roy H, Saul W-C, Wilson J (2016) Technical Report: Progress toward pathways prioritization in compliance to Aichi Target 9. Information documented presented at SBSTTA 20 UNEP/CBD/SBSTTA/20/INF/5, the twentieth meeting of the CBD’s Subsidiary Body on Scientific, Technical and Technological Advice, Montreal, Canada, 25–30 April 2016., 2–11.
  • Settele J, Hammen V, Hulme P, Karlson U, Klotz S, Kotarac M, Kunin W, Marion G, O’Connor M, Petanidou T, Peterson K, Potts S, Pritchard H, Pysek P, Rounsevell M, Spangenberg J, Steffan-Dewenter I, Sykes M, Vighi M, Zobel M, Kuhn I (2005) ALARM: Assessing LArge-scale environmental Risks for biodiversity with tested Methods. Gaia-Ecological Perspectives for Science and Society 14: 69–72. https://doi.org/10.14512/gaia.14.1.20
  • Shadish WR, Tolliver D, Gray M, Sengupta SK (1995) Author judgments about works they cite - 3 studies from psychology journals. Social Studies of Science 25: 477–498. https://doi.org/10.1177/030631295025003003
  • van Wilgen BW, Wilson JR (2018) The status of biological invasions and their management in South Africa in 2017. South African National Biodiversity Institute, Kirstenbosch and DST-NRF Centre of Excellence for Invasion Biology, Stellenbosch, 398 pp.
  • Volery L, Blackburn TM, Bertolino S, Evans T, Genovesi P, Kumschick S, Roy HE, Smith KG, Bacher S (2020) Improving the Environmental Impact Classification for Alien Taxa (EICAT): a summary of revisions to the framework and guidelines. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 547–567. https://doi.org/10.3897/neobiota.62.52723
  • Wilson JRU, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (2020) Frameworks used in invasion science: progress and prospects. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM (Eds) Frameworks used in Invasion Science. NeoBiota 62: 1–30. https://doi.org/10.3897/neobiota.62.58738
  • Wilson JRU, Faulkner KT, Rahlao SJ, Richardson DM, Zengeya TA, van Wilgen BW (2018) Indicators for monitoring biological invasions at a national level. Journal of Applied Ecology 55: 2612–2620. https://doi.org/10.1111/1365-2664.13251
  • Wilson JRU, García-Díaz P, Cassey P, Richardson DM, Pyšek P, Blackburn TM (2016) Biological invasions and natural colonisations are different – the need for invasion science. Neobiota 31: 87–98. https://doi.org/10.3897/neobiota.31.9185