Early detection of insect infestation is a key to the adoption of control measures appropriated to each local condition. The use of remote sensing was recommended for a quick scanning of large areas, although it does not work well with signals bearing low intensity or items that are difficult to detect. Unmanned Aerial Vehicle (UAV, or drone) may help in getting closer to individual trees and detect atypical signals of small dimensions. The larvae of the pine processionary moth (PPM, Thaumetopoea pityocampa (Denis & Schiffermüller, 1775, Lepidoptera, Notodontidae) build conspicuous silk nests on the external parts of the host plants at the beginning of the winter and their early detection may prompt managers to adopt management techniques. This work aims at testing two deep learning methods (Region-based Convolutional Neural Network - R-CNN and You Only Look Once - YOLO) to detect the nests under three different conditions of host plant species and forest stands in southern Europe. YOLO algorithm provided better results and it allowed us to achieve F1-scores as high as 0.826 and 0.696 for the detection of presence / absence and the individual nests, respectively. The detection of all the nests that can be present on a tree is not achievable with either UAV scanning or traditional ground observation, therefore the integration of the methods may allow the complete efficiency of the surveillance. The use of UAV combined with Artificial Intelligence (AI) image analysis is recommended for further use in forest and urban settings for the detection of the PPM nests. The recommended methods can be extended to other pest systems, especially when specific symptoms can be associated with an insect pest species.

AI algorithm, forest, Insecta, Lepidoptera, Notodontidae, pest, PPM, UAV, urban

The use of remote sensing can provide evidence of abnormal biological activity in forest ecosystems (Forzieri et al. 2021). New satellite constellations with more frequent flyovers and multispectral cameras were used to provide accurate images of forest cover and to better detect isolated patches of tree mortality (Senf et al. 2017; Sebald et al. 2021) and infer putative responsible biotic factors based on spatiotemporal dynamics patterns (Senf et al. 2015). Remote sensing techniques are efficient to detect crown discoloration of mature trees or patches of killed trees as well as temporary defoliation events (Sangüesa-Barreda et al. 2014). However, the images used so far still have coarse spatial resolution (40–50 cm) to detect damage within an individual tree and even less on parts of a tree. These very localised anomalies are low intensity and are difficult to detect at far distances and that is the main advantage of using Unmanned Aerial Vehicle (UAV, or drone), i.e., to get closer to individual trees to scan and detect atypical signals of small dimensions.

Two recent systematic reviews underline the growing use of UAV for forest health surveys (Duarte et al. 2022; Ecke et al. 2022) and address applications for mapping tree defoliation, and trees damaged by pine wood nematode and bark beetles. Specific assessments of individual tree defoliation caused by the pine processionary moth (PPM, Thaumetopoea pityocampa, Lepidoptera: Notodontidae) were done in Spain with an accuracy of about 80% (Cardil et al. 2017, 2019). Otsu et al. (2018, 2019) combined UVA image acquisition with novel image classification techniques to achieve 95% overall accuracy in detecting defoliation. The detection of individual PPM winter nests, however, was not attempted before our study. Winter nests start to be visible in the outer parts of the trees as soon as the larvae moult to the third instar (Uemura et al. 2021) and before significant damage is caused. Early detection would therefore be especially useful to predict tree defoliation and health risk to humans and domestic animals and thus suggest when to apply control methods preventively (Battisti et al. 2017). As long as the PPM is expanding to upper latitudes and higher elevations because of climate change (Roques 2015), an early detection – early action system should be the most recommended method to adopt, especially in urban forests that are next to be infested and where the perception of the risk by citizens and managers is not yet of high concern.

Beyond the acquisition of images by UAVs, the detection of objects on these images is mainly limited by the performance of analytical tools. The primary objective of this study was to compare different deep learning algorithms to meet the challenge of accurately counting objects in UAV images, such as the PPM winter nests that can be partially hidden in the tree canopy and may show blurred contours. With Region-based Convolutional Neural Network (R-CNN) algorithms (Nugroho 2018), significant improvements were achieved in the most important computer vision problems such as segmentation and object detection. The latest updated version of this deep learning network is Faster R-CNN (Ren et al. 2015). You Only Look Once (YOLO) algorithm is another widely used deep learning system for real-time object detection and is considered much faster than the previous one (Redmon et al. 2016). Comparing both models, Wu et al. (2021) demonstrated similar accuracy but higher running speed for YOLO in the frame of pine wilt disease surveys with UAVs.

Because our goal was to propose a method applicable in different conditions of development of PPM infestations, we conducted our studies in three sites in France, Italy and Portugal differing by the nature of the terrain and climatic conditions. The sites also allowed to test three major host-plants of PPM in the Mediterranean region, i.e., Pinus nigra Arnold, P. pinaster Aiton, and P. pinea L., which are characterised by different crown architecture. In doing so, we were also able to quantitatively test the performance of PPM nest detection in relation 1) to nest size (small vs large) and location in the tree crowns (periphery vs centre) than for nests located at the top of the tree crowns, and 2) to decreasing density of pine trees in the stand.

Study sites and ground assessment

Three study sites were selected in south-western Europe (France, Italy, and Portugal) to ensure a maximum of variability of conditions (Fig. 1). In Portugal (Arez - Alcácer do Sal, 38.315665°N, 8.493764°W, 39 m), we selected one pure even-aged Pinus pinea stands of 12 years old with a density 100 trees/ha (tree height 0.82–6 m, tree diameter at breast height (dbh) 9 cm). In France (Cestas, 44.779308°N, 00.795404°W; 61 m), we selected several Pinus pinaster plots in an even aged experimental plantation of 10 years old (ORPHEE), which had the advantage of comparing several stand densities (tree height 9.80 m, dbh 16 cm). The UAV survey was conducted on 4 blocks (c.a. 1.5 ha), each including 4 plots of 400 m² with a density of 2500, 1250, 825 and 625 trees/ha, respectively. In Italy (Lavini di Marco Trento, 45.8467°N, 11.0365°E, 680 m), we selected a site composed mainly by Pinus nigra in a natural uneven aged forest in the Southern Alps, growing on steep slopes with several isolated and relatively short trees (mean density of about 100 trees/ha, tree height 1.5–10 m, tree dbh 10–30 cm). It was thus less dense than the plantation forests.

Figure 1.

Location of the study sites and types of habitat a stand of Pinus pinea in Portugal b stand of Pinus pinaster in France c stand of Pinus nigra in Italy.

All three sites were surveyed for visual abundance inventory of PPM nests from the ground (two observers looking at both sides of the trees) and each tree was identified and geo-localised (Suppl. material 1: fig. S1). In addition to ground counts, on the French site, PPM nests were visually counted from a mobile platform (Fig. 1b) at 2 m above canopy of all trees of the 16 sampled plots. Several characteristics were also recorded such as the position on terminal shoot vs. lateral branch and the size of the nest by distinguishing between small and lightly woven nest (grey colour, weave not much beyond the needle clusters) vs. medium to large and well woven nest (white colour, weave enveloping the needle clusters).

A total of 936 trees were inventoried at the three sites, simultaneously from the ground and from UAV images, and they showed considerable differences in the rate of colonization. A total of 665 PPM nests were visually inventoried from the ground over the entire study and 222 nests were detected by human eyes on UAV images of the same trees (Table 1).

A total of 22,904 images composed the UAV database leading to 2,858 nests being visually assessed on the images due to multiple views of the same nest.

The performance of AI model architectures (FRCNN vs YOLO) was compared with human interpretation of UAV images for all images gathered on all trees from the three countries. This dataset included all trees counted from UAV images, and not only trees observed from the ground. A total of 1,477 trees were inventoried on UAV images (803 in France, 459 in Italy and 215 in Portugal). This dataset was used for comparing human visual interpretation of UAV images with AI models estimates, considering both presence of nests and their number per tree. YOLO architecture performed better than FRCNN with similar precision but better recall (less omission) and thus higher F1-score. Similar results were obtained for the presence of nests and the number of nests per tree (Fig. 2).

Figure 2.

Performance of FRCNN and YOLO architectures for the detection of (a) presence / absence of PPM nest and (b) number of PPM nests per tree using the full dataset of 1,477 observed trees on UAV images in France, Italy, and Portugal.

Using data from each country (i.e., different pine species) as replicates, we found significantly better F1-scores with YOLO than with FRCNN for both presence of nests (paired t-test, p = 0.02) and number of nests per tree (paired t-test, p = 0.03) (Fig. 3).

Figure 3.

F1-score of FRCNN and YOLO architectures for the detection of (a) presence / absence of PPM nest and (b) number of PPM nests per tree using the full dataset of 1,634 observed trees on UAV images in France, Italy, and Portugal.

The use of YOLO algorithm to identify the number of nests per tree detected from the ground provided results that did not differ significantly from those obtained with human eye interpretation of UAV images (paired t-test using countries as replicates, p = 0.97). The mean F1-scores were 0.238 and 0.242 for YOLO and human eye, respectively, suggesting low performance of both methods. However, the F1 score was three-fold higher for the detection of infested trees, irrespective of the number of nests, with F1-scores of 0.648 (YOLO) and 0.676 (human eye), respectively.

When nest detection from the ground was combined with nest detection from a platform (803 trees in 16 plots, French site), the tree infestation rate was 23% for ground and 19% for platform observations. YOLO performed similarly (paired t-tests, n = 16, p = 0.08) to detect the number of nests from the ground or from the platform, with mean F1 scores of 0.432 and 0.361, respectively. The same was observed for presence of nests (p = 0.11), with mean F1 scores of 0.526 and 0.438, respectively.

The performance of the YOLO algorithm was not significantly influenced by the density of maritime pine trees in the French site (ANOVA, n = 4, p = 0.83 and 0.56 for the number of nests and their presence, respectively), although the worst performance was obtained at the highest pine density (2500 trees/ha), where the canopy cover was 100% and the estimated percentage of infested trees the lowest (13%) (Fig. 4).

Figure 4.

F1-score of YOLO for the number of PPM nests per tree and their detection according to tree density per plot (400 m²) in the French site. Tree density corresponded to 2500, 1250, 825 and 625 trees/ha for the 100, 50, 33 and 25 trees per plot, respectively.

The performance of the algorithm was significantly influenced by the size of the nests (paired t-test, P = 0.008). The performance was not significantly influenced by the position of the nest (paired t-test, P = 0.442). The algorithm was performant at detecting the presence of nests more than 10 cm, irrespective of their localization on the terminal or lateral branches in the tree crown (Fig. 5).

Figure 5.

F1-score of YOLO for the detection of the presence of PPM nests according to the size of the nest (small < 10 cm diameter; big > 10 cm diameter) and position on the branch in the French site.

The use of AI proved effective to detect the nests of PPM on trees of different species and sizes, even under variable densities. In particular, the YOLO algorithm was superior to R-CNN for this special application. This result did not allow an exhaustive detection of the nests occurring on trees. The study proved the advantage of using UAVs to document the presence of at least one nest per tree. It therefore represents a substantial step forward in the integration of the UAV survey with ground observations in the monitoring of the colonies of an important forest defoliating insect in the Mediterranean area. Furthermore, the study paves the way for early detection of symptoms associated with the presence of pests and pathogens on the canopy of forest and ornamental trees, which is essential to elicit specific and targeted management measures.

The use of remote sensing in the detection of biotic disturbances was implemented for achieving higher performance of surveillance and for addressing management measures (Lehmann et al. 2015; Hall et al. 2016). The latter case is especially true for pathogens and pests that may cause defoliation or discoloration of a group of trees (Duarte et al. 2022; Ecke et al. 2022) or even individual trees in a stand of trees (Näsi et al. 2018) or in an urban setting (Wagner and Egerer 2022). The winter nest of PPM is a special target for UAV detection. In this species, the silk is spun as long as the temperature is decreasing at the beginning of the winter (Uemura et al. 2021), well before the massive defoliation occurs and becomes detectable (Battisti et al. 2015). Tracking early nest formation may thus allow managers to decide which control measures can be adopted among the few available under the different growing conditions of the trees (Roques 2015). As expected, the detectability of the presence of the PPM increased with the size of nests and thus, as nest volume increases exponentially during the fall and winter (Branco et al. 2008), the period of image acquisition will be a relevant variable to analyse.

The AI analysis performed equally well with different host-pine species, percentage of infested trees, and local topography. The YOLO algorithm always yielded satisfactory results in maximising the detection power of nests. Even when compared with the human eye’s careful inspection of each image, the YOLO algorithm performed equally well in identifying the trees carrying at least one PPM nest. The performance was different, however, when image data from UAVs were compared with ground/platform assessment of nest presence, which, of course, allows many more directions of observation than the one from above. The number of nests per tree counted from the ground often differed from the number of nests counted on the images, either by human eye or YOLO algorithm. It could be explained by a general underestimation or simply by counting different nests from the ground and from the UAV images taken from above. Overall, the quick flyover of a UAV over a forest stand or a city park largely outweighs the cost of detailed observations from the ground/platform, and in any case, the detection of nests from the UAV can inform people of the risk and the need to carry out more precise observations on the ground. Interestingly, the detection power was not affected by the stand density in the French site, except at the highest density of 2500 trees/ha for which tree crowns were overlapping. In contrast, nest size results to be the most important trait for detection.

As PPM is increasingly becoming a species of concern for forests and trees in relation to the rapid range expansion and large population growth in the areas of infested trees (Backe et al. 2021), the availability of a quick canopy scanning that can detect the early occurrence of nests seems to be a promising tool for pest managers, as shown for invasive alien species in forests (de Groot et al. 2020). With the refinement of small symptom detection from aerial images, especially when the contrast with the background is not as bright as in PPM, the method is potentially applicable to many organisms causing discoloration in tree canopies.

In conclusion, we demonstrate the potential use of IA on UAV images to detect at the tree level the presence of localised pests. Results significantly differ depending on IA algorithms, opening possibilities for further improvement. This technique can pave new avenues in the surveillance and management of emerging and non-native pests of trees, where early detection and early action should go together to achieve a satisfactory level of protection.

The study was conducted in the European project HOMED (Holistic management of emerging forest pests and diseases), which receives funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 771271. The team from the University of Lisbon was also funded by the Forest Research Centre, a research unit funded by Fundação para a Ciência e a Tecnologia I.P. (FCT), Portugal (UIBD/00239/2020). We thank Yannick Mellerin, Laurent Saléra and Rémy Dourthe for their help in the field work and Ricardo Cipriano for the access to pine stands. We thank INRAE - UEFP (Forest experimental Facility UEFP-https://doi.org/10.15454/1.5483264699193726E12) for the management of the ORPHEE site. We warmly thank two reviewers for comments on the manuscript.