The impact of a research publication is often measured by where it is published and how often it is cited (Biagioli 2016). By aggregating across publications, metrics have been developed to provide a measure of the impact of individual scientists and institutions (Hirsch 2005) that is incorporated into decisions around recruitment, promotions, and research funding (Hicks et al. 2015). While such metrics are simple and transparent, they create perverse incentives. For example, researchers, in an attempt to increase their h-scores, might inappropriately or egregiously promote their own work when reviewing or editing other people’s manuscripts (Biagioli 2016; Zaggl 2017). Nonetheless, and acknowledging that impact as measured by citations is a different concept from research quality (Bornmann and Haunschild 2017), citations are a useful starting point to evaluate impact.

We explored four main aspects. First, we assessed the proportion of self-citations to gauge the degree to which the frameworks were only used by those who constructed them. Second, we evaluated whether the geographic and taxonomic biases apparent in the scientific literature in general (cf. Wilson et al. 2007; Wuestman et al. 2019) and invasion science in particular (Pyšek et al. 2006; Pyšek et al. 2008) were also apparent in the papers citing the frameworks. Our expectation was that the selected frameworks would be used across taxa as they were designed to be generally applicable. For example, an explicit rationale for the development of the Unified Framework was to merge a scheme predominately used by zoologists (Williamson and Fitter 1996) with a scheme used predominately by botanists (Richardson et al. 2000). Third, we wanted to explore whether the citing papers actually implemented the frameworks or simply cited the papers to back up general comments about biological invasions. And finally, we wanted to assess the degree to which the citations were from studies focussing on biological invasions or whether the frameworks had impact beyond their originally-intended field of study.

We downloaded bibliographic information from the ISI Web of Science Core Collection (https://www.webofknowledge.com) on 1 July 2019 for all the publications listed as citing one of the four papers considered here (Blackburn et al. 2014; Blackburn et al. 2011; Hawkins et al. 2015; Hulme et al. 2008), and obtained copies of the citing publications if possible (books and book chapters were omitted if a digital copy could not be readily obtained – 3.2, 1.6, 2.8% of cases for the Pathway Classification, the Unified Framework, and EICAT respectively; Suppl. material 1). We developed an initial protocol to score the articles according to set criteria. Ten of the authors scored 10 papers to look at consistency in scoring (i.e., inter-rater reliability). For most categories, it was found to be consistent, but in a few cases (e.g., the discipline), we found there was some disagreement that could be reduced by refining the protocol. However, when attempting to score papers in terms of the degree of influence the frameworks had on the paper there was substantial disagreement, even after discussion to refine the categories [Fleiss’ Kappa of 0.179 in R package irr (Gamer et al. 2019)]. As a result, the extent of influence of each framework was scored by only one person for consistency (JRUW scored the Pathway Classification and SK scored the Unified Framework and EICAT, after discussing and aligning the scoring categories, see Suppl. material 2: Table S2.1). Most authors did some scoring of the other sections. We then adapted the protocol (see Suppl. material 2.1) and scored each paper accordingly (see Table 1 for details of the data extracted). It took 2–10 minutes to score each paper once it was downloaded.

The list of journals that cited each framework was extracted. To determine whether the frameworks had impact beyond their originally-intended field of study, we assigned each citing journal to one of three categories – those that explicitly included biological invasions as a subject area; those that published other aspects of ecology or were more general in scope; and those that did not include ecology as a subject area.

To evaluate geographic biases in the papers citing the frameworks, we used the results of a previous analysis of the geographic pattern of invasion science as a whole (Pyšek et al. 2008) as a point of comparison. We identified the corresponding author of studies that had cited the frameworks and assigned their primary affiliation to a geographic region as per the regions used by Pyšek et al. (2008). We then compared the number of studies in each region relative to the number of studies noted in Pyšek et al. (2008) against the expectation based on the rest of the world. After adjusting for multiple comparisons, regions that tended to have cited one of the frameworks either more often or less often than expected were identified (see Suppl. material 2.5).

Because it was difficult to be sure how the frameworks had influenced publications, we surveyed the corresponding authors of papers that cited any one of the four papers. The survey was conducted under ethical clearance (SU project number: 14445) issued by Stellenbosch University.

The questionnaire (Suppl. material 2.2) was structured to assess how the frameworks are viewed and why they were cited in the authors’ works. We used structured questions that were adopted and modified from a framework that has been applied to survey authors in citation analyses (Case and Higgins 2000; Harwood 2008; Prabha 1983; Shadish et al. 1995). The questionnaire consisted of 35 questions in four sections: eight proximity questions that assess the relationship between the person who cited a publication and any authors of that publication; 21 questions that seek reasons why authors might cite a paper; four semi-structured questions to gauge whether the frameworks are used in research or to implement policy and management strategies; and two questions that provided an opportunity to list any suggestions for or proposed improvements to the frameworks (Suppl. material 2.2). The questions that elicit reasons why authors might cite a paper can be grouped into five broad citation categories – classic citations, negative citations, creative citations, personal influence citations, and supportive citations – and the results were interpreted in the context of these groupings. A cover letter and a link to the questionnaire were emailed to a total of 958 corresponding authors, with a reminder sent to non-responders after one week. The survey ran for three weeks, from 13 March to 6 April 2020.

Policy papers and strategies, unlike journal articles, often do not have a comprehensive list of references, are not indexed by academic databases, and many are published in languages other than English. Therefore, we read a selection of national and international policy documents. These documents included national strategies, status reports, national and international guidelines, and documents published by the Convention on Biological Diversity, International Union for Conservation of Nature, and the European Union. We then qualitatively assessed the degree to which the documents explicitly or implicitly referred to or implemented the frameworks. For this purpose, we only considered documents dated more recently than 2008, i.e., after the Pathway Classification was published.

The results of the citation analysis are summarised in Table 1. As of 1 July 2019, the Pathway Classification had 436 citations recorded on the ISI Web of Science database, the Unified Framework 729 citations, and the two papers that present and refine EICAT 249 citations. This puts them in the top ten most cited papers in their respective journals amongst articles published in the same year or more recently. The vast majority of these citations are from papers that can be classified as invasion science. In fact, about a third of all papers published in the journal Biological Invasions in 2018 cite the Unified Framework. The numbers of citations are increasing annually, with no indication of any plateaus (Figure 1). The number of self-citations has also increased over time, but their relative proportion has declined. Twelve percent of the papers cited more than one of the frameworks (Suppl. material 2.3).

Figure 1.

The number of times each framework paper was cited since publication until 1 July 2019. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (\Blackburn et al. 2014; Hawkins et al. 2015). Values from 2019 only include a portion of the year and even the number of citations by articles published in 2018 is a slight underestimate as it has also increased in the time since July 2019. Each framework has shown a general increase in citations per year since publication, and a decrease in the proportion of self-citations (Table 1).

Almost half of the citing papers only cited the frameworks to justify general comments about biological invasions. Importantly, however, the citing papers covered a wide range of realms and taxa, and the frameworks were implemented in detail in a similar wide range of studies (Figure 2).

Figure 2.

The extent to which the frameworks have influenced citing papers broken down by (a) environment and (b) taxonomic groups. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; Hawkins et al. 2015). The widths of the bars are proportional to the number of citations. The degree to which the framework was used in the citing paper increases from left to right on each figure [from general, to definition, to broad (application), to specific (application)]. The data are in Suppl. material 1, and the methodology used for scoring in Suppl. material 2.1.

The frameworks were cited by articles published in a wide range of journals (151, 223, and 108 journals, see Suppl. material 2.4). Unsurprisingly, the majority of these journals (70, 68, and 79%) have invasions as one of or their main subject area. Similarly, the majority of articles citing each framework (83, 85, and 87%) were explicitly on invasion science. All three frameworks have a global reach and have been cited by authors from around the world working on invasions in a similar global range of sites (Suppl. material 2.5). However, when compared with the analysis of citation patterns in invasion biology (Pyšek et al. 2008), all the frameworks are more frequently cited by researchers based in Europe or South Africa and less often by those based in North America [49, 44, and 52% of all citations to the respective frameworks were from research led by European based authors vs. 22% of all studies in Pyšek et al. (2008); for South Africa: 9, 16, 18% vs. 2%; for North America: 20, 17, 12% vs. 50%; (the probability from a Chi-squared test was < 0.01 in all these cases)]. See Suppl. material 2.5 for the full details.

We received responses from 84 people contacted (~ 9% of the 905 e-mails that did not bounce) from 20 countries (including 14 responses from North America, a slight over-representation). Responses were split fairly evenly across the frameworks (20 of 256; 51 of 589, and 13 of 113 respectively). The statement that respondents tended to most agree with was “This reference is authored by recognized authorities in the field” followed by “This is a classic reference in the field”. The most common reason for citing the frameworks was that they are “classic citations” (Fig. 3, Shadish et al. 1995). Importantly, of the six statements that suggest a paper is viewed as a “classic citation”, the two questions that were not widely supported (in fact more respondents disagreed than agreed) were “There have been substantial efforts to show that the framework is wrong” and “The framework has withstood many efforts to show that it is wrong”. Therefore, while the papers are undoubtedly viewed and used as classic citations, there is a general feeling that the frameworks have not been adequately investigated. This was borne out by various suggestions of how the frameworks could (and in some cases have) been modified or where other frameworks are more appropriate (Suppl. material 2.6).

Figure 3.

Reasons for citing the frameworks based on the response to a questionnaire sent to corresponding authors. The frameworks are the Pathway Classification (Hulme et al. 2008), the Unified Framework (Blackburn et al. 2011), and EICAT (Blackburn et al. 2014; Hawkins et al. 2015). The categories Negative, Supportive, Personal Influence, Creative, and Classic are based on Shadish et al. (1995). See Suppl. material 2.2 for a copy of the questionnaire and Suppl. material 2.6 for the full results and how the questions map on to different categories.

In terms of the link between the citing authors and the authors of the frameworks, over half have spoken to one of the authors (64, 51, 95%) and a substantial number of these consider one of the authors a personal friend (35, 13, 41%). While the respondents often recommended the citation to others during review (40, 27, 46%), it was not suggested to them often (5, 6, 0%). Of course, the respondents are a small section of the invasion science community who have actively cited the framework and who were willing to respond to a survey concerning the framework.

All three frameworks seem to have had some impact on policy and management (Suppl. material 2.6 and 2.7; Box 1). The Pathway Classification framework has arguably had the most impact. The CBD’s Aichi Biodiversity Target 9 specifies (amongst other things) the need to identify and prioritise pathways by 2020 and a modification of the pathway framework was proposed for use by the CBD itself (Scalera et al. 2016). Other examples of its adoption include the guidelines for invasive species planning and management on islands published by IUCN and the European Union Regulation on the prevention and management of the introduction and spread of invasive alien species. More recently, EICAT has been adopted as an IUCN standard (IUCN 2020) and is anticipated to play an important role in future biodiversity targets and as part of an indicator to track impact (Essl et al. 2020; GEO BON 2015; Latombe et al. 2017). By contrast, we found little evidence that the Unified Framework (the most cited framework investigated here) has been used in policy and management. This could in part be due to differences in how the phenomenon of biological invasions is defined (Wilson et al. 2016). Policy-orientated definitions of invasive organisms often include the impact of the organism, while the biogeographic definition also represented in the Unified Framework (Richardson et al. 2000) seems to be more common in science.

Importantly, once a scientific framework has been widely accepted by an international body like the CBD or the IUCN, it is very likely that the original references are no longer cited. Therefore, caution must be observed in interpreting the policy influence of scientific documents.

The Unified Framework and EICAT owe substantial intellectual debts to previous papers and frameworks. Indeed, some of the original frameworks are arguably still more influential. The Unified Framework is based partly on a framework for plants outlined by Richardson et al. (2000), and this earlier paper still tends to be more widely cited. Richardson et al. (2000) had a huge effect on the study of biological invasions. By creating standards that were widely adopted by the research community, data on biological invasions have been increasingly based on a common set of criteria, and are therefore directly comparable. This has facilitated a wide range of comparative analyses [e.g., the Global Naturalized Alien Flora (GloNAF) project (Pyšek et al. 2017)].

Frameworks also evolve and develop over time and in some cases are superseded. The Pathway Classification has been expanded and subcategories developed as part of its proposed uptake by the CBD (Harrower et al. 2017; Scalera et al. 2016). It is noticeable that some more recent journal articles and policy documents implement the CBD pathway classification scheme without citing the original paper on which it is based. This might be quite typical, i.e., once a framework is adopted into a policy or adapted into a guideline, there is a step-change in the impact it has, but conversely, the original paper might no longer be cited. Papers applying EICAT often use it in combination with the Generic Impact Scoring System (GISS; Nentwig et al. 2016; Nentwig et al. 2010) upon which EICAT is based or they use a modification using aspects of both schemes. GISS has probably been more often applied to date and to a wider taxonomic range than EICAT (e.g., Kumschick et al. 2015), but due to EICAT’s adoption as an IUCN Standard, EICAT is rapidly gaining momentum.

Frameworks often need to be adapted in light of practical experience. For example, several adaptations to the Unified Framework have been proposed based on experiences of implementing it in Europe (Groom et al. 2019), Hawaii (Brock and Daehler 2020), and South Africa (Wilson et al. 2018). Similarly, Pergl et al. (2020) and Faulkner et al. (2020) provide proposals to refine the CBD pathway classification scheme based on applying it in different contexts, and Volery et al. (2020) document changes made to EICAT after stakeholder consultation. All three frameworks have, to different extents, been incorporated in developing biodiversity data standards, and this will provide a more formalised process for revising them.

Importantly, however, our results show that the extent of influence of the frameworks is still somewhat affected by how they were originally developed. There is, unsurprisingly, a high level of self-citations, and this likely explains part of the apparent European and South African bias in uptake (cf. Fig. 1 and Suppl. material 2.5). The Pathway Classification was a direct product of the European Union Funded ALARM project (Settele et al. 2005); the idea to develop the Unified Framework arose at a workshop in Switzerland and was further elaborated at a meeting in South Africa; and EICAT resulted from a workshop in Germany. Moreover, of the 33 original authors of the frameworks, 26 are based in Europe, three in South Africa, and one each in Australia, Canada, New Zealand, and the USA. In this context, the global influence of the frameworks has been impressive, but it will be important for the utility of the frameworks to be assessed in more depth and in different contexts. For example, introduction pathways have changed over time (Faulkner et al. 2016; Hulme 2009), and the importance of different pathways varies across the world. Faulkner et al. (2020) highlight one such case, where the trade in traditional medicines is a potentially important introduction pathway in Africa, but is not considered explicitly in the current (arguably Eurocentric) Pathway Classification. Similarly, the Unified Framework originated from combing zoological and botanical frameworks, and there are several practical issues applying both it and the Pathway Classification to fungi and microbes (Paap et al. 2020).

Nonetheless we believe that our results provide some indication that invasion science is beginning to coalesce around systematic schema for classification and understanding that are applicable across taxa and realms.

Given the lack of points of comparison, it is difficult to gauge whether the results seen here are surprising or not. Ideally, we would have looked at the uptake of other important frameworks in invasion science (including historical and more contemporary schemes) and compared with highly-cited framework papers from related disciplines. Unfortunately, the methodology we developed was time-consuming. We found no reasonable proxy for a manual analysis of the extent of influence of the frameworks on the citing papers. There was a broad correlation between our manual scoring of the extent of influence and the number of times a framework was cited, but there were many exceptions (Table 1). Similarly, simply noting whether a citation was in the methods, discussion, or introduction provided some indication of whether the frameworks were used, but not enough to reliably predict that the frameworks were actually implemented (results not shown, but data presented in Suppl. material 1). We concluded that conducting such a citation analysis requires careful examination of at least the sentences that include the citation, and often an evaluation of the whole manuscript. Machine-learning techniques might offer a solution to this issue in future.

We did, however, identify some comparisons that would be particularly interesting and some important research gaps. As mentioned previously, most of the frameworks presented here had progenitors [for example, the Unified Framework explicitly built upon Williamson and Fitter (1996) and Richardson et al. (2000)], an explicit evaluation of how these have been used over time would provide a benchmark against which our results could be assessed. Similarly, while the frameworks chosen reflect pathways, species, and impacts, it would be important to consider frameworks centred around sites of invasion or the effectiveness of interventions (McGeoch et al. 2016; Wilson et al. 2018), or to consider how invasion hypotheses are cited (Catford et al. 2009; Jeschke and Heger 2018). Finally, it might be instructive to track recent frameworks [e.g., SEICAT, the socio-economic impact classification of alien taxa scheme (Bacher et al. 2018), although there has not been much time for uptake].

Our research did not primarily set out to evaluate citation practices, but several insights were apparent. Many of the citations were what we considered frivolous (and informally dubbed “citation fluff”). The introduction of most papers starts with a generic catch-all statement about invasions, and the frameworks were often used to support these, often inappropriately [e.g., citing the Unified Framework as evidence that invasions have impact, or EICAT as a risk assessment protocol (Kumschick et al. 2020-a)]. Arguably “citation fluff” provides an indication of influence, i.e., the frameworks are not directly and explicitly used, but play a role in shaping the overall mental model of the processes at play. However, there were many errors in the way in which the frameworks are cited (see Suppl. material 2.3). Should there be a greater onus on authors, reviewers, and editors to purge “citation fluff” or at least to ensure such references really support the general statements made?

The fact that 20–40% of all citations are self-citations is not necessarily indicative of nepotistic or insular research practices (Seeber et al. 2019). The frameworks were the products of highly productive scientists with the intention of producing seminal papers in a research field in which they were amongst the research leaders. The rapid uptake and declining proportion of self-citations are arguably, healthy signs, as is the geographic spread of the citations. This is borne out by the respondents to the survey where the authors were considered well respected, and the framework papers were, by and large, considered classic papers in the field.

However, the papers analysed and the people surveyed were very biased. The results are, therefore, consistent with the notion of a distinct school of thought amongst certain (particularly European and South African) invasion scientists for whom these frameworks are valuable (cf. the MAFIA framework of Pyšek et al. 2020). A study of researchers who did not cite or use these frameworks despite the framework being relevant to (or even designed to assist) their research would do much to further our understanding of the limitations of the frameworks. As an analogue, it is difficult to understand why some invasions are successful if we lack data on failed invasions (Zenni and Nuñez 2013).

Finally, as a side note, in our experience papers with numeric citations are harder to read, comment on as editors and reviewers, and make analyses, like the one here, much more cumbersome. It is not clear to us why online-only publishers (e.g. the Public Library of Science) persist with this format (cf. https://svpow.com/2011/01/07/an-open-letter-to-plos-one-a-pox-on-your-numbered-references/).