Corresponding author: John R. U. Wilson ( john.wilson2@gmail.com ) Academic editor: Julie Lockwood
© 2020 John R. U. Wilson, Arunava Datta, Heidi Hirsch, Jan-Hendrik Keet, Tumeka Mbobo, Khensani V. Nkuna, Mlungele M. Nsikani, Petr Pyšek, David M. Richardson, Tsungai A. Zengeya, Sabrina Kumschick.
This is an open access article distributed under the terms of the Creative Commons Attribution License (CC BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Citation:
Wilson JRU, Datta A, Hirsch H, Keet J-H, Mbobo T, Nkuna KV, Nsikani MM, Pyšek P, Richardson DM, Zengeya TA, Kumschick S (2020) Is invasion science moving towards agreed standards? The influence of selected frameworks. In: Wilson JR, Bacher S, Daehler CC, Groom QJ, Kumschick S, Lockwood JL, Robinson TB, Zengeya TA, Richardson DM. NeoBiota 62: 569-590. https://doi.org/10.3897/neobiota.62.53243
|
The need to understand and manage biological invasions has driven the development of frameworks to circumscribe, classify, and elucidate aspects of the phenomenon. But how influential have these frameworks really been? To test this, we evaluated the impact of a pathway classification framework, a framework focussing on the introduction-naturalisation-invasion continuum, and two papers that outline an impact classification framework. We analysed how these framework papers are cited and by whom, conducted a survey to determine why people have cited the frameworks, and explored the degree to which the frameworks are implemented. The four papers outlining these frameworks are amongst the most-cited in their respective journals, are highly regarded in the field, and are already seen as citation classics (although citations are overwhelmingly within the field of invasion science). The number of citations to the frameworks has increased over time, and, while a significant proportion of these are self-citations (20–40%), this rate is decreasing. The frameworks were cited by studies conducted and authored by researchers from across the world. However, relative to a previous citation analysis of invasion science as a whole, the frameworks are particularly used in Europe and South Africa and less so in North America. There is an increasing number of examples of uptake into invasion policy and management (e.g., the pathway classification framework has been adapted and adopted into EU legislation and CBD targets, and the impact classification framework has been adopted by the IUCN). However, we found that few of the citing papers (6–8%) specifically implemented or interrogated the frameworks; roughly half of all citations might be viewed as frivolous (“citation fluff”); there were several clear cases of erroneous citation; and some survey respondents felt that they have not been rigorously tested yet.
Although our analyses suggest that invasion science is moving towards a more systematic and standardised approach to recording invasions and their impacts, it appears that the proposed standards are still not applied consistently. For this to be achieved, we argue that frameworks in invasion science need to be revised or adapted to particular contexts in response to the needs and experiences of users (e.g., so they are relevant to pathologists, plant ecologists, and practitioners), the standards should be easier to apply in practice (e.g., through the development of guidelines for management), and there should be incentives for their usage (e.g., recognition for completing an EICAT assessment).
Biological invasions, EICAT, introduction pathways, invasion science, Pathway Classification, Unified Framework
The field of invasion science has grown rapidly (
These issues were discussed as part of a workshop on “Frameworks in Invasion Science” in November 2019 (
All three frameworks–the Pathway Classification ( Status report on biological invasion in South Africa South African regulations on biological invasions require that, every three years, a report on the status of biological invasions and the effectiveness of control measures and regulations is produced. The primary aim of the status report is to strengthen the links between basic research, policy, and management by detailing the current status and providing support to decision-makers. The first report was released in October 2018 and it was the first effort globally to report on the status of biological invasion at a national level ( Risk analysis framework The South African regulatory lists ( |
Due to the way we selected the three frameworks, our analysis is somewhat circular. For example, the frameworks were selected on the basis that there has been some uptake into policy, so it is unsurprising that we found some uptake by policy-makers. However, we feel it is important to: (i) establish whether these frameworks are used broadly by people interested in invasion science or used just by a subset (e.g., only researchers based in Europe or only people studying marine invasions); (ii) determine whether the frameworks are being used as they were intended or only used to justify working on biological invasions; (iii) to assess how users perceive the frameworks; and (iv) to draw insights on how the field could move forward.
To evaluate the impact of the frameworks, we conducted an analysis of the citations of the papers, surveyed the authors of citing papers, and explored the extent to which the frameworks have been used in policy and management documents.
The impact of a research publication is often measured by where it is published and how often it is cited (
We explored four main aspects. First, we assessed the proportion of self-citations to gauge the degree to which the frameworks were only used by those who constructed them. Second, we evaluated whether the geographic and taxonomic biases apparent in the scientific literature in general (cf.
We downloaded bibliographic information from the ISI Web of Science Core Collection (https://www.webofknowledge.com) on 1 July 2019 for all the publications listed as citing one of the four papers considered here (
Summary of the results of the citation analysis of frameworks in invasion science. Where numbers are given, they are for the Pathway Classification framework (
Variable | Type | Description | Expectation | Results |
---|---|---|---|---|
Discipline | Factor with three levels (invasion, ecology, other) | A rough indication of what the topic of the paper is. | No specific expectation, but provides an indication of the extent to which the frameworks have been used beyond invasion science. | The vast majority of citing papers were directly related to biological invasions (96, 93, 92%), but all of the frameworks were cited by some broader ecological (or evolutionary) papers (4, 6, 7%), and a handful of papers in other disciplines (< 1% in each case, including some in journals with apparently no link to ecology, for example, the American Journal of Roentgenology). |
Extent of influence | Ordered factor with four levels (general, definition, broad, specific) | An interpretation of how the citation is actually used (i.e., the degree to which the paper implements the framework). This provided a response variable for testing other variables against. | Papers citing the frameworks should tend to implement specific aspects of the proposed frameworks, although they might also be used to make general points about biological invasions. | The level of frivolous citations was surprisingly high ~ 50% of all citations were classed as general (i.e. “citation fluff”), with only 6–8% of citing papers actually implementing the frameworks. This pattern was similar across the frameworks studied here. (see Figure |
Self-citation | Factor with two levels (TRUE, FALSE) | Whether authors of the original paper were also authors of the citing article. | There should be a significant number of self-citations, but this should decline over-time as other people start using the framework. Authors of the framework would be more likely to specifically utilise the framework. | Confirmed (Figure |
There were a large number of self-citations, (25, 22, 39%), but the relative proportion of self-citation is declining over time (generalised linear model with binomial errors, with year as an explanatory variable and whether a reference was a self-citation as the response using Chi-squared test of the change of variance: p = 0.02; p < 0.01; p = 0.11). | ||||
Self-citations were more likely to implement the frameworks in detail than use them as “citation fluff” (analysis based on an ordered factor of the extent of influence versus self-citation: F3,417 = 6.1, p < 0.01; F3,708 = 4.0, p < 0.01; F3,240 = 4.8, p < 0.01). | ||||
Realm | Factor with six levels (freshwater, marine, terrestrial, other, multiple, NA) | The scope of the paper in terms of the environment. For laboratory-based studies, this is based on the taxa used. | A core rationale for the development of the frameworks was to make them generalisable across different environments (cf. |
Largely confirmed, but with notable biases (Figure |
All three frameworks are cited by studies across different realms and taxa, though most were terrestrial studies and most on animals. However, it is notable that, when implemented in detail, the Pathway Classification was particularly used for comparisons across taxa and realms (i.e. multiple), while the Unified Framework tended to focus on particular realms (freshwater or terrestrial) and particular taxa (animals or plants) rather than for comparisons. | ||||
Taxon | Factor with five levels (animals, plants, other, multiple, NA) | The taxonomic scope of the organisms studied in the paper. | A core rationale for the development of the frameworks was to make them generalisable across different taxa. As such, the expectation was the frameworks are used consistently across taxa. | Largely confirmed, but with notable biases (Figure |
See results for realms above. Studies on animals that cited EICAT tended to have implemented the framework more than other studies and there were no fungal or microbial studies as yet. | ||||
Number of citations | Integer | The number of times the framework is cited in the paper. | Papers that cite a framework multiple times would be more influenced by those frameworks. | Confirmed, but there is a large amount of variation. |
The extent of influence tended to increase with number of citations, but this factor on its own did not explain much of the variance in number of citations (31, 23, 30%); and some papers that specifically implemented the frameworks only cited the frameworks once, while other papers that used the frameworks as “citation fluff” still had numerous citations (maximums of 4, 9, 5). In conclusion, the amount of variance explained was not enough to suggest that the number of times the framework is cited in text is a reliable proxy for the extent of influence of the framework on a paper. |
The list of journals that cited each framework was extracted. To determine whether the frameworks had impact beyond their originally-intended field of study, we assigned each citing journal to one of three categories – those that explicitly included biological invasions as a subject area; those that published other aspects of ecology or were more general in scope; and those that did not include ecology as a subject area.
To evaluate geographic biases in the papers citing the frameworks, we used the results of a previous analysis of the geographic pattern of invasion science as a whole (
Because it was difficult to be sure how the frameworks had influenced publications, we surveyed the corresponding authors of papers that cited any one of the four papers. The survey was conducted under ethical clearance (SU project number: 14445) issued by Stellenbosch University.
The questionnaire (Suppl. material
Policy papers and strategies, unlike journal articles, often do not have a comprehensive list of references, are not indexed by academic databases, and many are published in languages other than English. Therefore, we read a selection of national and international policy documents. These documents included national strategies, status reports, national and international guidelines, and documents published by the Convention on Biological Diversity, International Union for Conservation of Nature, and the European Union. We then qualitatively assessed the degree to which the documents explicitly or implicitly referred to or implemented the frameworks. For this purpose, we only considered documents dated more recently than 2008, i.e., after the Pathway Classification was published.
The results of the citation analysis are summarised in Table
The number of times each framework paper was cited since publication until 1 July 2019. The frameworks are the Pathway Classification (
Almost half of the citing papers only cited the frameworks to justify general comments about biological invasions. Importantly, however, the citing papers covered a wide range of realms and taxa, and the frameworks were implemented in detail in a similar wide range of studies (Figure
The extent to which the frameworks have influenced citing papers broken down by (a) environment and (b) taxonomic groups. The frameworks are the Pathway Classification (
The frameworks were cited by articles published in a wide range of journals (151, 223, and 108 journals, see Suppl. material
We received responses from 84 people contacted (~ 9% of the 905 e-mails that did not bounce) from 20 countries (including 14 responses from North America, a slight over-representation). Responses were split fairly evenly across the frameworks (20 of 256; 51 of 589, and 13 of 113 respectively). The statement that respondents tended to most agree with was “This reference is authored by recognized authorities in the field” followed by “This is a classic reference in the field”. The most common reason for citing the frameworks was that they are “classic citations” (Fig.
Reasons for citing the frameworks based on the response to a questionnaire sent to corresponding authors. The frameworks are the Pathway Classification (
In terms of the link between the citing authors and the authors of the frameworks, over half have spoken to one of the authors (64, 51, 95%) and a substantial number of these consider one of the authors a personal friend (35, 13, 41%). While the respondents often recommended the citation to others during review (40, 27, 46%), it was not suggested to them often (5, 6, 0%). Of course, the respondents are a small section of the invasion science community who have actively cited the framework and who were willing to respond to a survey concerning the framework.
All three frameworks seem to have had some impact on policy and management (Suppl. material
Importantly, once a scientific framework has been widely accepted by an international body like the CBD or the IUCN, it is very likely that the original references are no longer cited. Therefore, caution must be observed in interpreting the policy influence of scientific documents.
We found that the invasion frameworks assessed here are widely cited by studies focussing on different realms and taxa, and from many different parts of the world. While many citations might be viewed as frivolous (“citation fluff”), there is a substantial number where the frameworks have been implemented in detail. There are different possible explanations for these trends. Invasion science might be coalescing temporarily; it might be settling down to adopt standard and widely-agreed practices; a particular ‘school’ of invasion science that uses particular frameworks might be emerging; or there might always be a suite of papers that are core papers for citing, but that do not actually influence the direction of the field. We discuss some of these issues here.
The Unified Framework and EICAT owe substantial intellectual debts to previous papers and frameworks. Indeed, some of the original frameworks are arguably still more influential. The Unified Framework is based partly on a framework for plants outlined by
Frameworks also evolve and develop over time and in some cases are superseded. The Pathway Classification has been expanded and subcategories developed as part of its proposed uptake by the CBD (
Frameworks often need to be adapted in light of practical experience. For example, several adaptations to the Unified Framework have been proposed based on experiences of implementing it in Europe (
Importantly, however, our results show that the extent of influence of the frameworks is still somewhat affected by how they were originally developed. There is, unsurprisingly, a high level of self-citations, and this likely explains part of the apparent European and South African bias in uptake (cf. Fig.
Nonetheless we believe that our results provide some indication that invasion science is beginning to coalesce around systematic schema for classification and understanding that are applicable across taxa and realms.
Given the lack of points of comparison, it is difficult to gauge whether the results seen here are surprising or not. Ideally, we would have looked at the uptake of other important frameworks in invasion science (including historical and more contemporary schemes) and compared with highly-cited framework papers from related disciplines. Unfortunately, the methodology we developed was time-consuming. We found no reasonable proxy for a manual analysis of the extent of influence of the frameworks on the citing papers. There was a broad correlation between our manual scoring of the extent of influence and the number of times a framework was cited, but there were many exceptions (Table
We did, however, identify some comparisons that would be particularly interesting and some important research gaps. As mentioned previously, most of the frameworks presented here had progenitors [for example, the Unified Framework explicitly built upon
Our research did not primarily set out to evaluate citation practices, but several insights were apparent. Many of the citations were what we considered frivolous (and informally dubbed “citation fluff”). The introduction of most papers starts with a generic catch-all statement about invasions, and the frameworks were often used to support these, often inappropriately [e.g., citing the Unified Framework as evidence that invasions have impact, or EICAT as a risk assessment protocol (
The fact that 20–40% of all citations are self-citations is not necessarily indicative of nepotistic or insular research practices (
However, the papers analysed and the people surveyed were very biased. The results are, therefore, consistent with the notion of a distinct school of thought amongst certain (particularly European and South African) invasion scientists for whom these frameworks are valuable (cf. the MAFIA framework of
Finally, as a side note, in our experience papers with numeric citations are harder to read, comment on as editors and reviewers, and make analyses, like the one here, much more cumbersome. It is not clear to us why online-only publishers (e.g. the Public Library of Science) persist with this format (cf. https://svpow.com/2011/01/07/an-open-letter-to-plos-one-a-pox-on-your-numbered-references/).
The selected frameworks are influential and widely cited. They are being used to provide information about explicit efforts at monitoring and reporting biological invasions and the development of internationally-agreed data standards. Nonetheless, they are not yet widely implemented as they were originally formulated. We believe that our ability to understand and manage biological invasions will improve as we move increasingly towards agreed standards in the field (
This paper emerged from a workshop on ‘Frameworks used in Invasion Science’ hosted by the DSI-NRF Centre of Excellence for Invasion Biology in Stellenbosch, South Africa, 11–13 November 2019, that was supported by the National Research Foundation of South Africa and Stellenbosch University. Workshop participants are thanked for their insight on an earlier draft, Chris Delport and Tamara Robinson for insight into how to score papers, and Suzaan Kritizinger-Klopper for useful discussions. The South African Department of Forestry, Fisheries and the Environment, (DFFtE) are also thanked for funding noting that this publication does not necessarily represent the views or opinions of DFFtE or its employees. PP was supported by EXPRO grant no. 19-28807X (Czech Science Foundation) and long-term research development project RVO 67985939 (Czech Academy of Sciences). DMR acknowledges support from the Oppenheimer Memorial Trust (grant 18576/03). TAZ was supported by the DSI-NRF Centre of Excellence for Invasion Biology (CIB) and National Research Foundation (Grant 103602).