
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39232041
71074
10.1038/s41598-024-71074-2
Article
Urban traffic tiny object detection via attention and multi-scale feature driven in UAV-vision
Wang Yangyang
Zhang Jie Jiezhang_sy_sjt@163.com

Zhou Jian
https://ror.org/05ct4s596 grid.500274.4 Academy of Military Sciences, Institute of Systems Engineering, Beijing, 100000 China
4 9 2024
4 9 2024
2024
14 2061412 3 2024
23 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
The unmanned aerial vehicle (UAV) city patrol is of great significance in ensuring the safety of residents’ lives and properties, as well as maintaining the normal operation of the city. However, the detection of UAV images faces challenges such as numerous small-scale objects, complex backgrounds, and high requirements for detection speed. In response to these issues, we introduce a Real-time Small Object Detection network in UAV-vision (RTS-Net), tailored for UAV patrols. Initially, we introduce a multiscale feature fusion module (MFFM) designed to augment the expressiveness of features across scales, thereby enhancing the detection of smaller objects. Subsequently, leveraging attention mechanisms, we present the coordinated attention detection module (CADM), which bolsters the detection model’s ability to accurately segregate objects from the background in expansive, complex scenarios. Lastly, a lightweight real-time feature extraction module (RFEM) is crafted to diminish model computational complexity and boost inference speed. On the UAV road patrol image dataset we constructed, our proposed method attains a detection accuracy of 89.9% mAP, breaking previous records. It surpasses all prevailing detection methods, particularly for small-scale objects. Simultaneously, it achieves an inference speed of 163.9 FPS. The experimental results show that RTS-Net can satisfy the accurate and efficient detection of ground objects by various different UAV platforms in different complex scenarios.

Subject terms

Electrical and electronic engineering
Engineering
Mathematics and computing
Computational science
Computer science
Information technology
Software
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

With the rapid development of unmanned aerial vehicle (UAV) technology, its reliability and endurance have been greatly enhanced. Thanks to its flexible flight capabilities and cost advantages, UAV is favored by people and widely used in scenarios such as logistics transportation, agricultural crop protection, disaster relief, providing great convenience to people’s lives. As a novel means of acquiring spatial data, UAV aerial photography technology is beginning to be widely applied in tasks such as urban road patrols. Compared to traditional manual patrolling, UAV can not only save a lot of manpower and material resources, but also will be unaffected by ground-based challenges like traffic congestion, painting a promising future for their use. However, the successful operation of UAV within cities largely hinges on stable data connections. UAV must transmit raw images to ground stations in real-time, which after human analysis, lead to subsequent operational commands. This not only burdens the operators with intensive data interpretation tasks but also poses severe challenges to the bandwidth and stability of the data connections. Any disruptions in connectivity can impair UAV operations, even leading to potential damages, thereby limiting their widespread adoption for urban patrols1–3.

In recent years, deep learning-based object detection technology has achieved remarkable outcomes, and the competing methods have shown unparalleled precision and extensive generalizability in diverse scenarios, which makes it possible for UAV to avoid the dependence on the network and realize the automatic acquisition of semantic information of urban road images by piggybacking on object detection methods. Regrettably, the majority of contemporary object detection algorithms have been designed for natural scene images, and the detection of UAV patrol images will also face the following challenges.UAV images contain more small objects that are difficult to identify. When the UAV is performing patrol tasks, it is typically far from the ground, causing ground objects within its field of view to appear fine-grained. These small-scale objects not only lack visual features, making it difficult for the object detection network to extract sufficient useful information, but also, as the depth of the neural network increases and the size of the feature maps decreases, these small objects may face the risk of feature disappearance, leading to serious missed detections.

The scenes in UAV images are very complex. Influenced by urban environments, weather, lighting, and other variations, the scenes in UAV images can be very complex and changeable. These complex backgrounds severely interfere with the features of the objects to be recognized, making it difficult for the object detection network to effectively separate objects from the background, resulting in serious false detections and missed detections.

UAV patrols impose higher requirements on the speed of object detection. UAV typically fly at high speeds during patrols, requiring the detection network to accurately identify ground objects in a short time. Otherwise, large areas may go unrecognized by the patrol UAV, posing potential safety hazards for the city.

In response to these challenges, Khan et al.4 proposed a two-stage high-resolution image detection method, effectively improving the model’s multi-scale object detection capability by adding multiple region proposal networks to convert each level of the feature pyramid into independent multi-scale proposal generators. Yang et al.5 proposed predicting coarse locations on low-resolution features first, and then calculating accurate results on high-resolution features based on these coarse locations. Hoanh et al.6 combined the Transformer encoder-decoder structure with the object-focus network and applied it to the multi-layer feature maps of the feature pyramid, significantly enhancing the detection accuracy for high-resolution images. Additionally,3,7–14 also proposed their methods for improving small object detection. However, these methods are often dedicated to improving detection accuracy, while sacrificing detection speed to a certain extent. Therefore, they cannot fully meet the real needs of UAV urban patrol.

In this paper, we introduce a real-time small object detection method for UAV patrolling, named RTS-Net. First, we analyze the challenges faced in small object detection and design a novel feature fusion module as the network neck for object detection.By fully utilizing shallow high-resolution features in the backbone network, this module enhances the feature representation capability for small-sized objects, thereby improving the accuracy of small object detection. Secondly, to distinguish objects against intricate backdrops, we’ve developed a detection head using attention mechanisms, allowing UAV to hone in on areas of interest in their wide visual range. Finally, we implemented a lightweight bottleneck module as the primary structure for the main network and feature fusion, aiming to reduce computational and parametric overhead, bolstering inference speeds without compromising detection accuracy significantly. Experimental results on our constructed dataset named DRP-Dataset (Drone Road Patrol Dataset) demonstrate that RTS-Net can meet the high-precision real-time detection of ground objects during UAV patrolling.

Contributions are summarized as bellow:We analyzed the challenges faced in detecting small objects and proposed a new feature fusion module, enhancing the feature representation capabilities of small objects, thereby improving UAV detection accuracy.

We designed an attention detection head based on attention mechanisms, enhancing the UAV’s ability to focus on attention regions amidst complex backgrounds, thereby bolstering the reliability of urban UAV patrols.

We constructed a UAV image dataset based on the characteristics of urban UAV patrols, and through practical experiments, we demonstrated that the method proposed in this paper can effectively enhance the detection performance of UAV images while ensuring real-time processing.

The rest of this paper is organized as follows. Section “Related work” discusses related works. Section “Method” details the proposed method. Section “Experiments” provides an analysis of the experiments and results. Finally, Sect. “Conclusion” concludes the paper.

Related work

Current advancements in the domain of UAV image object recognition can predominantly be categorized into traditional object detection methods and deep learning-based object detection techniques.

Traditional object detection techniques for UAV

Traditional object detection approaches usually harness manually crafted features for object feature extraction, followed by the utilization of classifiers to solve object identification and classification within specific UAV imaging scenarios. Numerous researchers have endeavored to adapt these conventional methodologies for UAV-based image object detection. Gaszczak et al.15 proposed generating a thermal map of UAV imagery first, and then deploying a cascaded classifier in conjunction with multi-variate Gaussian shape matching for vehicle detection. While this method boasts rapid detection speeds, its accuracy in complex backgrounds is limited, rendering it unsuitable for real-world applications. Sokalski et al.16 introduced an approach that constructs saliency maps for UAV images by extracting various feature maps, such as the mean deviation map, histogram-enhanced contrast saliency map, and multi-channel color feature map. However, this approach is also inadequate in complex backdrops. Cao et al.17 , aiming to enhance the detection efficacy in varied scenarios, introduced an augmented histogram of gradient feature to better characterize object shapes and appearances. They employed a Linear Support Vector Machines (SVM) classifier to detect vehicles within UAV imagery, achieving improved detection under diverse lighting conditions, albeit at a relatively slower inference speed. Moranduzzo et al.18 utilized SVM to categorize the Scale-Invariant Feature Transform (SIFT) keypoints in UAV images into vehicle and non-vehicle classes, enhancing the precision of ground vehicle detection. Nevertheless, extracting SIFT features is computationally expensive. This same author later presented a sliding window search approach for extracting HOG features from images in another study19. Subsequently, a dictionary lookup was used for feature point classification, substantially accelerating the detection process. Su et al.20 described a method that extracts the Histogram of Oriented Gradients (HOG) features from UAV images to detect ground vehicles. This approach initially sets a small sliding window on the input image, calculates the HOG features of this windowed area to estimate the primary gradient direction of the vehicle, rotates the sliding window based on this estimated direction, and employs four integral histograms to compute the HOG features within the window. Finally, an online cascaded boosting algorithm is utilized for vehicle recognition. While it effectively alleviates sample imbalance issues and delivers commendable detection results, it exhibits significant false positives with objects like rooftops and windows that bear resemblance to vehicles. In addition, methods21–24 have also basically achieved object detection in UAV images.

The aforementioned methods, which rely on handcrafted feature extraction, have achieved ground vehicle detection from UAV. However, the extraction of manual features necessitates ample prior knowledge and is heavily reliant on the designer’s expertise. The inability to adaptively extract features and the challenge of extracting high-level semantic information from UAV images with diverse lighting, weather, and intricate urban road environments result in poor scene transferability. Thus, crafting a universal detection approach to cater to extreme UAV detection scenarios remains elusive. In contrast, deep learning-based object detection techniques possess adaptive feature extraction capabilities and deliver higher accuracy in ground object detection, showcasing significant potential for applications in complex urban environments.

Deep learning-based approaches

Chen et al.25 recognized the limitations imposed by handcrafted feature extraction on object detection performance. Consequently, they adopted Deep Neural Networks (DNN) to extract rich features from images. They proposed a modification to the DNN by replacing the final convolutional layer and the max-pooling layer with modules having multiple receptive fields. This change facilitated the extraction of multi-scale features from images, leading to substantial improvements in vehicle detection. Bejiga et al.26 leveraged a linear SVM to classify features extracted using Convolutional Neural Networks (CNN), enhancing the precision of detecting individuals in UAV imagery. Addressing the issue of high-resolution in UAV images, Ammour et al.27 suggested segmenting the image into smaller scales before feeding them into a CNN for feature extraction. The features were then classified using an SVM, enhancing the accuracy of ground vehicle detection. Bazi et al.28 also proposed a Convolutional Support Vector Machine (CSVM) based on CNN and the SVM classifier. This approach effectively detected objects in UAV imagery, especially when training samples were limited. However, these methods primarily relied on CNN for feature extraction followed by SVM for classification, rather than integrating them into a cohesive network structure. As a result, neither the detection speed nor precision met the demands of UAV in real-world scenarios.

Today, various end-to-end deep convolutional networks have achieved exhilarating results in the domain of object detection. These methodologies unify feature extraction, object classification, and localization within a single comprehensive network, significantly augmenting detection performance29–39. Vaddi et al.40 extended the foundational architecture of RetinaNet by incorporating ResNet50 and MobileNet as the backbone feature extractors. This adaptation yielded commendable results for UAV imagery detection but grappled with issues like redundant anchor creation and suboptimal detection of obscured objects. Zhang et al.41 enhanced the Yolov3 detection network by pruning its structure, leading to the more computationally efficient SlimYolov3. This modification substantially reduced computational complexity, boosting inference speeds, yet faced challenges with lower detection rates. Addressing the prevalent challenge of detecting small-scale objects in UAV imagery, Zhu et al.42 augmented the Yolov5 with an additional small object detection head, which enhanced the precision for such objects. However, the resultant computational complexity was significantly high, leading to slower detection speeds. Aimed at the practical application of object detection algorithms in UAV scenarios, Cao et al.43 designed a lightweight multi-scale object detection method by stacking two lightweight modules they developed. While ensuring precision, this method reduced model size and computational costs. Implemented on an embedded processor Jetson Xavier NX, it achieved an inference speed of 100 FPS. Nonetheless, due to the limited diversity in the experimental datasets, the model’s robustness across varying illumination and weather conditions in urban scenarios remains to be further verified. Ye et al.44, confronting challenges faced during UAV urban patrols, designed a detection method integrating CNN and Transformer. This fusion achieved monumental advancements in detection precision and inference speed. Considering the stringent reliability and stability demands of UAV urban patrols, the generalizability and detection precision of this method warrant further enhancement. Du et al.45 developed a context-enhanced group normalization (CE-GN) layer that replaces statistics based on sparse sampling features with global context statistics, and then designed an adaptive multi-layer masking strategy to generate the optimal mask ratio at different scales for compact foreground coverage, thereby improving the detection accuracy of small objects in UAV images.

In summation, current deep learning-based UAV image object detection techniques have achieved significant strides in detection precision and inference speeds. Yet, they still grapple with issues like low detection accuracy, slow inference speeds, and suboptimal generalizability. These challenges hinder their full potential in consistently and reliably patrolling the ever-evolving urban landscapes. Therefore, this study, cognizant of the challenges and practical necessities of UAV urban patrolling, introduces a real-time small object detection methodology. This approach ensures rapid inference while significantly enhancing the recognition precision of typical ground objects within UAV imagery.

Method

We introduce a real-time small object detection network for UAV during urban patrols, termed RTS-Net. This architecture primarily consists of three modules: a Multiscale Feature Fusion Module (MFFM), a Coordinated Attention Detection Module (CADM), and a Real-time Feature Extraction Module (RFEM). Specifically, the MFFM enhances detection accuracy for small-scale objects by intensively mining detailed characteristics from the underlying features. The CADM amplifies the ability to focus on objects within intricate backgrounds by scanning regions of interest in the feature maps. The RFEM, designed with a lightweight structure, reduces the computational intricacy of the backbone network, thus accelerating the speed of inference.Fig. 1 RTS-Net architecture overview.

Overview of RTS-Net

The overall architecture of the proposed RTS-Net is depicted in Fig. 1. It mainly comprises the input section, backbone feature extraction network, feature fusion module, and detection head. Operating as a single-stage object detector, it necessitates only a single forward pass to predict the class and positional data of the object directly. Within the input section of RTS-Net, an interpolation algorithm is employed to resize images of arbitrary dimensions to a fixed resolution of 960×960, serving as input for the backbone network. During training, data augmentation techniques like random color jittering and mosaic are utilized. In the backbone feature extraction network, input features undergo downsampling by a factor of four through two convolution layers. This reduces the computational demand and simultaneously extracts preliminary shallow features. Following this, the RFEM and convolution layers, as part of a deep neural network, continuously downsample features while progressively abstracting higher-level semantic information. Towards the end of the backbone, a Spatial Pyramid Pooling-Fast (SPPF) module is integrated. It establishes multi-branch, multi-scale pooling layers to create and amalgamate features of varied scales, thereby enhancing the network’s multi-scale feature representation capacity. Subsequently, the MFFM facilitates interaction and fusion among features at different levels in the backbone. This supplies shallow features with the advanced semantic intel from deeper features and complements deep features with rich details from the shallow layers, refining detection accuracy across multi-scale objects, especially diminutive ones. In the detection head, CADM will be employed to extract regions of interest containing the objects in the images, followed by the detection of objects at various scales across feature maps with different resolutions. Experimental results showcase that RTS-Net achieves exceptional detection accuracy and inference speed across diverse UAV imaging scenarios.

Real-time feature extraction module, RFEM

The rapid flight speed of UAV causes swift changes in the images captured from their view. As a result, it necessitates that the object detection algorithms swiftly recognize these images. To minimize computational demands and accelerate inference speeds, we introduce a streamlined and effective RFEM. RFEM segregates high-dimensional feature maps along the channel dimension, subsequently employing multi-branch convolutions for parallel computation on these segmented feature maps, thereby enhancing computational efficiency. Ablation experiments show that the use of this module can significantly reduce the model’s computational load by 26.5% and increase the inference speed by 30.9%, without significantly affecting detection accuracy.Fig. 2 The architecture of RFEM.

The architecture of RFEM is illustrated in Fig. 2. For the input features, a 1×1 convolution is first used to adjust its channel number. It is then split by channel dimension into two low-dimensional features. These are further divided into blocks by averaging the channel count and processed in parallel using a 3×3 convolution. Subsequently, these processed segmented features are concatenated with the initial features and passed through a 1×1 convolution for feature fusion and channel count adjustment. The equation can be represented as:1 y1,y2=split(conν1(x))

2 y=BN(conν2(concat(y1,∑i=1Cφi(y1),y2)))

In the formula, x represents the input feature, y signifies the output feature,conv1 denotes the standard 1×1 convolution, y1 and y2 symbolize the two intermediate features that are obtained by performing the split operation and segmenting them on average along the channel dimension. φi(·) represents the operation of segmenting the feature into the lower-dimensional features, and all φi(·) have the same structure and are parallelly computed, reducing complexity and boosting efficiency. C represents the count of multi-branch structures, with C=16, in this paper. Batch normalization (BN) aids model learning.

Utilizing RFEM to construct the backbone feature extraction network significantly reduces the model’s overall parameter and computational burdens, heightening the inference speed of the detection algorithm. Empirical evaluations indicate that RFEM does not compromise the detection precision substantially, fostering an optimal equilibrium between detection accuracy and inference velocity.

Multiscale feature fusion module, MFFM

In deep neural networks, as the network depth increases, while more advanced semantic information from the image is extracted, the decline in resolution also results in the gradual loss of intricate detail within the image. Due to the trade-off between high resolution and high-level semantic information, achieving high-precision detection for small objects has always posed a significant challenge. Typically, shallower feature maps possess a higher resolution, capturing more of the image’s textural, appearance, and other intricate features, as well as object positional information - making them more conducive for small object detection and localization. In contrast, deeper feature maps, with their reduced resolution, have a larger receptive field and carry richer high-level semantic information, making them more suitable for the detection and classification of larger objects46–51.Fig. 3 The architecture of MFFM.

Fig. 4 Feature weighted fusion.

To capitalize on the diverse information available in features from different layers and enhance the network’s multi-scale detection precision (especially for small objects), we introduce the MFFM, as depicted in Fig. 3. This module allows the network to employ features from various hierarchical levels to predict objects of differing scales. Furthermore, via shortcut connections, it channels high-level semantic information beneficial for object classification from deeper features to the shallower ones and supplements the deeper features with positional information vital for object localization. Distinct from the feature fusion module in common object detection methods, MFFM is particularly attentive to the plethora of minute objects in UAV imagery. As such, it harnesses more superficial features from the backbone network, termed P1in, and offers subsequent detection heads a high-resolution feature, P1out, more apt for small object detection. Experiments affirm the efficacy of this approach in enhancing the network’s precision for small object detection. Additionally, we acknowledge that features from different levels contribute variably to the prediction of objects of various scales. Directly blending features from diverse layers could potentially undermine the expressive capability of features apt for different scale object detection. As a remedy, drawing inspiration from the Bidirectional Feature Pyramid Network (BiFPN), we architected a weighted fusion of features. As illustrated in Fig. 4, features stemming from different locations, when fused, are weighted by a learnable coefficient, thereby bolstering the expressive power of features tailored for the detection of objects at that particular scale. The governing equation for this is:3 XOutput=∑iwi·XInputi∑iwi+ε

In the given equation, XInputi denotes the input features originating from varying levels, while XOutput signifies the fused output features. wi stands for the learnable fusion weights, and 0≤wi≤1 with ϵ being a small positive value, serves to circumvent numerical instability. Moreover, MFFM integrates a direct connection channel between the input and output for every hierarchical level, enriching feature fusion without incurring additional costs. Specifically, its dual feature fusion can be expressed as:4 Pitm=Conν(ReLU(w0·Piin+w1·Resize(Pi+1in)w0+w1+ε))

5 Piout=Conv(ReLU(w0′·Piin+w1′·Pitm+w2′·Resize(Pi-1out)w0′+w1′+w2′+ε))

Here, Piin represents the input features of the i layer within MFFM, Pitm signifies the intermediate features of the top-down i layer in MFFM, and Piout designates the output features of the i layer in MFFM. Resize alludes to up-sampling or down-sampling operations to adjust the resolution, while Conv typifies a standard 1×1 convolution. After the feature fusion is completed, we further employ the RFEM module in MFFM to process the fused features, aiming to enhance the representation of features at different scales at a relatively low cost. This step aims to provide a better foundation for subsequent detection tasks.

Coordinated attention detection module, CADM

UAV tasked with urban patrolling typically come equipped with high-resolution cameras and fly at significant altitudes, giving them a vast field of view over cityscapes. This expansive viewpoint, combined with the intricate urban environment, often yields images with complex, perplexing background scenarios. Such a broad field of view and intricate backgrounds not only pose challenges for object detection algorithms in accurately localizing objects but also discerning between objects and backgrounds, leading to significant false positives and false negatives. To bolster the robustness of the algorithm, we introduce the Coordinated Attention Detection Module52 (CADM) ahead of the detection head, equipped with a Coordinate Attention mechanism. This module directs the model to focus on pertinent objects and their respective locations by extracting attention-centric regions from the image.Fig. 5 Illustration of coordinated attention.

Coordinate attention is a lightweight yet efficient attention mechanism. Unlike the Squeeze-and-Excitation (SE) attention53, it contemplates both inter-channel dependencies and the significance of spatial features for accurate object localization. When juxtaposed with the Convolutional Block Attention Module (CBAM)54, it circumvents the limitation of convolutions capturing only local correlations, enabling a broader spatial search for attention regions, thereby enhancing robustness in intricate backgrounds. As illustrated in Fig. 5, to prevent the loss of positional information through global average pooling, it is disintegrated into two parallel one-dimensional feature encoding operations. Specifically, for input feature X∈RC×H×W pooling (H, 1) kernels of size and (1, W) respectively encode along the horizontal and vertical directions for every channel, producing output features Zh∈RC×H×1 and Zw∈RC×1×W. Consequently, the output for the cth channel with a height h of is:6 zch(h)=1W∑0≤i<Wxc(h,i)

Similarly, the output of the cth channel with width w is:7 zcw(w)=1H∑0≤j<Hxc(j,w)

These transformations independently aggregate features along two spatial directions, yielding two direction-aware feature maps. They encapsulate long-range dependencies captured by the input feature map along one spatial direction and precise positional information along the other, aiding the object detection model in more accurately pinpointing regions of interest.

To effectively harness inter-channel dependencies and fully utilize the captured positional information, attention regions are subsequently generated based on these perception feature maps. Specifically, dimension transformation and concatenation are applied to the two feature maps, resulting in a concatenated feature Zc∈RC×1×(H+W) . 1x1 convolution refines this to produce a downsampled feature Zc′∈RC/r×1×(H+W) , expressed as:8 f=δ(F1([zh,zw]))

Where [·,·] denotes concatenation along spatial dimensions, F1 is a standard 1x1 convolution for downsampling, δ is a nonlinear activation function, f∈RC/r×1×(H+W) represents intermediate feature maps encoding spatial information horizontally and vertically, and r parallels the downsampling rate in SE attention, with r=32 in this context. f is subsequently split into two distinct vectors, fh∈RC/r×H and fw∈RC/r×W , along spatial dimensions. Two 1x1 convolutions, Fh and Fw , then respectively transform fh and fw to match the channel count of the input feature X . This can be formulated as:9 gh=σ(Fh(fh))

10 gw=σ(Fw(fw))

Where σ is the Sigmoid activation function. Finally, gh and gw are expanded and used as attention weights, weighed against input feature X . Consequently, the output of Coordinate Attention can be expressed as:11 yc(i,j)=xc(i,j)×gch(i)×gcw(j)

For feature maps of varying scales, CADM initially employs Coordinate Attention to derive the image’s attention map. This map then refines the input features via weighting, following which predictions are made for object classes and their positions within the image.

Experiments

Dataset preparation

Currently, the majority of public datasets used for object detection tasks are based on natural scenarios. Their feature distribution significantly differs from UAV images, making it challenging to accurately verify the effectiveness of proposed methods in UAV patrol scenarios. Therefore, for the real-world scenario of UAV road patrols, this paper constructs a dataset named DRP-Dataset consisting of 10,425 aerial images captured by UAV. The DRP-Dataset covers various UAV road patrol scenarios from different locations and times. We annotated the dataset with seven classes of typical ground objects, namely, people, bicycle, car, truck, tricycle, bus, and motor, according to the actual needs, with a total of 342,284 instances. Table 1 shows the statistical results of the number of instances of each class in the DRP-Dataset and the area occupied by objects of each class in the image. Among them, we use average area ratio (AAR) to evaluate the size of the object scale, which is expressed as the proportion of pixels occupied by the object in the image. From the results, we can see that most of the object instances in this dataset occupy less than 1% of the pixels in the image, which poses a relatively serious challenge to the detector’s ability to detect small objects. Hence, we categorize person, motor, and bicycle classes with an average area ratio of less than 0.2% as small objects, truck and bus classes with an average area ratio greater than 1.0% as large objects, and the remaining car and tricycle classes as medium-sized objects. We randomly split the dataset into training, validation, and test sets with a ratio of 6:2:2.Table 1 Number of instances in each class in DRP-Dataset.

Class	Bus	Truck	Car	Tricycle	
Instances	3946	11013	145171	4549	
AAR	2.23%	1.08%	0.51%	0.24%	
Class	Motor	Bicycle	Person	Total	
Instances	21763	32541	123301	342284	
AAR	0.11%	0.08%	0.07%	–	

In addition, to evaluate the detection performance of RTS-Net in more challenging scenarios and verify its generalization capability across diverse UAV platforms, we also conducted tests on the public VisDrone dataset55,56. VisDrone is an extremely challenging UAV-aerial image object detection dataset, comprising 10,209 static images of varying resolutions captured using different UAV platforms. These images span a variety of scenarios, weather conditions, and lighting situations and are annotated with 10 classes of typical ground objects like pedestrian, people, and bicycles, totaling around 2.6 million instances. We conducted experiments on RTS-Net based on the settings of the VisDrone dataset.

Implementation details

The experiments were conducted on a server equipped with an Intel®Core™ i9-13900K CPU and an NVIDIA GeForce RTX 4090 GPU, running the Ubuntu 18.04 LTS operating system. We implemented the proposed method using the deep learning framework PyTorch 1.13.0, with Python 3.9 as the programming language. The model was trained, validated, and tested using consistent hyperparameters. The optimizer employed was the stochastic gradient descent with momentum (SGDM). The initial learning rate was set at 0.01, with a momentum of 0.937 and a weight decay of 5×10-3. Training was performed for a maximum of 300 epochs. A warm-up strategy was employed during the first three epochs, and the learning rate was decayed by a factor of 0.1 every 100 epochs to enhance training stability.Fig. 6 Visual detection results of different methods.

Table 2 Comparison with state-of-the-art detectors on the DRP-Dataset. Param represents the number of parameters in the model, measured in (M). FLOPs represents the computational load of the model, measured in (G).

Method	Car	Person	Motor	Bicycle	Truck	Bus	Tricycle	mAP	Param	FLOPs	FPS	
Faster-RCNN	96.9	75.1	73.9	63.2	88.9	95.9	69.3	80.5	41.4	178	14.3	
Swin-transformer	96.9	78.8	75.9	71.7	90.0	96.3	73.2	83.3	44.8	183	14.8	
Yolov5s	93.1	70.9	72.3	67.4	86.2	85.6	73.2	78.4	7.2	16.5	212.8	
Yolov6s	96.7	61.8	70.9	54.7	87.4	95.6	60.5	75.3	18.5	45.3	138.1	
Yolov7-tiny	98.2	76.5	69.6	59.2	91.0	96.6	70.8	80.3	6.0	18.1	86.0	
Yolov8s	97.9	77.7	76.1	74.0	88.0	97.3	81.5	84.6	11.1	28.5	189.9	
DETR	91.8	33.1	57.4	34.4	86.0	94.6	52.4	64.2	41.6	81.6	4.0	
DDQ	97.5	84.2	77.5	68.6	91.6	96.3	70.9	83.8	-	-	14.2	
DINO	94.2	77.6	57.7	55.6	75.5	81.3	56.3	71.2	47.6	235	14.3	
EfficientNet	96.5	58.3	60.8	52.0	90.3	96.7	69.3	74.8	18.5	81.3	14.0	
RTS-Net	98.4	85.9	81.4	81.3	90.5	97.3	82.0	88.1	8.1	28.0	188.8	
RTS-Net(960×960)	98.1	88.5	86.2	86.8	91.4	96.2	82.0	89.9	8.1	28.0	163.9	

Comparisons with state-of-the-art detectors

We report the comparative results of our method against state-of-the-art object detection approaches on UAV road patrol images, as shown in Table 2. We use AP/mAP to evaluate the detection accuracy of the model and FPS to gauge its inference speed. The best results are highlighted in bold. Among them, Faster-RCNN30, Swin-Transformer57, DET58, DDQ59, DINO60 and EfficientNet61 are implemented based on mmdetection3.2.062, and we conducted experiments using their default parameters.

From the experimental results, it is evident that the proposed RTS-Net achieved the highest average detection accuracy of 89.9% mAP on the DRP-Dataset. This is 5.3% mAP higher than the second place Yolov8s and 6.1% mAP higher than the third place DDQ. It reached an inference speed of 163.9 FPS on an NVIDIA RTX 4090 GPU, which is 11 times faster than both Faster-RCNN and DDQ, meeting real-time detection requirements. Although it is slower than Yolov5s and Yolov8s, the average detection accuracy is 11.5% mAP and 5.3% mAP higher, respectively, making this trade-off in speed entirely justifiable. Moreover, the parameters of RTS-Net are only 8.1 M, and the computation is only 28.0 GFLOPs, making it one of the top methods in terms of efficiency. In summary, RTS-Net strikes an excellent balance between detection accuracy and inference speed, enhancing recognition performance on UAV images while ensuring real-time detection.

Furthermore, it is noteworthy that RTS-Net’s detection accuracy for small-scale object classes, namely person, bicycle, and motor, stands at 88.5%, 86.8%, and 86.2% AP respectively, far surpassing all other methods. This underscores RTS-Net’s superior small object detection capability, making it especially apt for UAV patrol scenarios where small objects are prevalent. In comparison to Yolov8s, the detection accuracies for these three classes improved by 10.8%, 12.8%, and 10.1% AP, suggesting that the techniques proposed in this paper effectively enhance the model’s ability to detect smaller objects. For medium-scale object classes, car and tricycle, RTS-Net also achieved detection accuracies of 98.1% and 82% AP respectively, marking a modest gain over Yolov8s.

To more vividly showcase the improvements RTS-Net brings to UAV city patrol image detection, we provide a qualitative analysis in Fig. 6. From left to right, each column sequentially displays the original image, the detection results of Yolov8s, and RTS-Net. For the purpose of ease of observation, we have removed class names and confidence level information, using rectangular boxes of different colors to represent objects of different classes. it can be observed that in Fig. 6a, Yolov8s not only missed a large number of bus objects under the overpass but also exhibited a relatively serious false detection phenomenon. In comparison, RTS-Net has much lower rates of both missed detections and false detections. In the bottom left corner of Fig. 6b and the bottom right corner of Fig. 6c, Yolov8s failed to recognize objects obstructed by buildings and trees. Additionally, in Fig. 6b, it could not effectively identify bicycles in front of the signboard. In contrast, RTS-Net demonstrated a more robust ability to resist complex backgrounds and obstructions. In Fig. 6d, RTS-Net exhibited significantly better small object detection capabilities compared to Yolov8s, identifying more motorcycle objects at the top of the image. Experimental results indicate that RTS-Net shows stronger capabilities in small object detection and resistance to complex backgrounds in UAV training images.

Moreover, in Fig. 7, we present the confusion matrix of the test results. The x-axis denotes the true class labels of the samples, while the y-axis represents the predicted outcomes. From the figure, it is apparent that RTS-Net boasts high recognition accuracy across all object classes. Both the false negative rate and the false positive rate are exceptionally low, ensuring the reliability of UAV-based urban patrols.Fig. 7 Confusion matrix.

Table 3 Comparison result on VisDrone.

Method	Pedestrian	People	Bicycle	Car	Van	Truck	Tricycle	Awning-Tricycle	Bus	Motor	mAP	
Faster-RCNN	37.0	30.3	18.2	74.0	45.7	38.6	27.3	14.2	54.7	41.4	38.1	
Swin-transformer	39.5	32.8	21.4	75.2	47.7	42.9	33.0	16.7	56.5	44.6	41.0	
Yolov5s	40.1	32.7	10.5	73.5	34.7	28.0	16.6	9.9	42.7	38.6	32.7	
Yolov6s	40.2	30.4	10.9	77.4	44.6	37.9	27.6	15.8	54.6	44.0	38.3	
Yolov7-tiny	40.9	37.3	9.4	77.3	38.3	30.1	21.8	10.8	46.3	44.6	35.7	
Yolov8s	42.2	32.9	14.8	79.3	45.5	36.9	28.2	16.3	58.9	45.9	40.1	
DETR	8.2	7.8	0.5	36.9	23.2	16.6	10.1	2.3	31.6	13.6	15.5	
DDQ	60.6	50.9	30.8	84.1	53.3	39.2	38.9	20.9	57.6	58.8	49.5	
DINO	58.0	45.3	21.5	83.5	49.1	36.3	32.5	17.1	52.6	55.1	45.1	
EfficientNet	20.8	17.2	5.6	63.3	27.2	29.1	11.8	6.6	32.0	24.1	23.8	
RTS-Net	61.8	49.2	25.3	87.3	55.5	43.9	35.9	21.1	66.5	60.0	50.7	

On the VisDrone dataset, we further evaluate the detection performance of RTS-Net in more challenging scenarios and its generalization ability on different UAV platforms. As depicted in Table 3, it can be observed that our proposed RTS-Net consistently achieves the highest average detection accuracy on the VisDrone dataset, registering 50.7% mAP. This is a notable 10.6% mAP improvement over Yolov8s and 1.2% mAP higher than DDQ. We believe this superior performance can be attributed to the abundance of small-scale objects present in the VisDrone dataset, and RTS-Net, leveraging its exceptional small object detection capability, significantly enhances the mean detection accuracy. Moreover, RTS-Net performs well in most classes, reinforcing its suitability for UAV imagery detection. In Fig. 8, we provide some visualized results of RTS-Net on the VisDrone dataset, further emphasizing RTS-Net’s robust generalization across various intricate scenarios and diverse UAV platforms.Fig. 8 Visualization results in VisDrone.

Fig. 9 Result of robustness testing.

Robustness testing

While patrolling with UAV, one inevitably encounters adverse weather conditions such as low light and haze, which impose high demands on the robustness of the detection model. Thus, we conducted experiments to test the resilience of RTS-Net under various challenging conditions. As can be seen from Fig. 9, RTS-Net can accurately identify ground objects even in adverse weather conditions such as haze, as shown in Fig. 9a, ensuring the UAV’s stable and reliable execution of patrol missions in various weather scenarios. Low-light conditions significantly degrade the imaging quality of cameras, but Fig. 9b indicates that RTS-Net can effectively resist the impact of low-light conditions on object detection, enabling the UAV to conduct all-weather patrols. Robustness tests demonstrate that RTS-Net can achieve high-precision detection of UAV images in various challenging environments.Table 4 Result of ablation experiments. Param represents the number of parameters in the model, measured in (M).

Model	MFFM	CADM	RFEM	Input_size	Param	FLOPs	FPS	mAP	
Baseline	–	–	–	640×640	11.1	28.5	189.9	84.6	
Baseline + MFFM	✓	–	–	640×640	11.7	38.1	158.7	87.9	
Baseline + CADM	–	✓	–	640×640	11.2	28.5	171.4	86.1	
Baseline + RFEM	–	✓	–	640×640	7.6	20.1	256.4	85.0	
Baseline + MFFM + CADM	✓	✓	–	640×640	11.7	38.1	144.2	88.6	
Baseline + CADM + RFEM	–	✓	✓	640×640	7.7	20.2	232.6	85.6	
Baseline + MFFM + RFEM	✓	–	✓	640×640	8.1	27.9	227.3	87.7	
RTS-Net	✓	✓	✓	640×640	8.1	28.0	188.8	88.1	
RTS-Net	✓	✓	✓	960×960	8.1	28.0	163.9	89.9	
FLOPs represents the computational load of the model, measured in (G).

Ablation studies

To further validate the efficacy of the proposed modules, we designed ablation studies, with results showcased in Table 4. Using Yolov8s as a baseline, the introduction of the MFFM module significantly boosted the detection accuracy by 3.3% mAP. However, the inference speed correspondingly decreased by 31.2 FPS, and the overall computation increased by 9.6 GFLOPs. After removing the MFFM module from RTS-Net, the detection accuracy decreased by 2.5% mAP, while the inference speed increased by 43.8 FPS, and the overall computation reduced by 7.8 GFLOPs. This indicates that while the MFFM module effectively improves the model’s detection accuracy, it has a certain impact on inference speed. Introducing the CADM module on the baseline improved the model’s detection accuracy by 1.5% mAP, with an 18.5 FPS decrease in inference speed and almost no change in overall computation. Removing the CADM module from RTS-Net resulted in a 0.4% mAP decrease in detection accuracy, a 15.3 FPS decrease in inference speed, and almost no change in overall computation. This demonstrates that the CADM module can improve the model’s detection accuracy without significantly affecting detection speed and overall computation. Finally, introducing the RFFM module to the baseline increased detection accuracy by 1.5% mAP, while inference speed improved by 66.5 FPS, and overall computation decreased by 8.4 GFLOPs. Removing the RFFM module from RTS-Net resulted in a 0.5% mAP increase in detection accuracy, a 44.6 FPS decrease in inference speed, and a 10.1 GFLOPs increase in overall computation. This indicates that the RFFM module can significantly improve the model’s detection speed without greatly impacting the model’s accuracy.

Furthermore, we recognized that increasing the resolution of the feature map can effectively enhance the model’s ability to recognize small objects. Therefore, we attempted to upscale the input image resolution from 640×640 to 960×960 to preserve more visual features of smaller objects. Experimental validation indicated that this strategy results in a 1.8% mAP improvement in detection accuracy, while the inference speed only drops by 24.9 FPS, still meeting real-time detection requirements. According to the experimental results of the last two rows in Table 2, increasing the input image resolution significantly improved the detection accuracy for small object classes such as person, motor, and bicycle, proving the effectiveness of this approach.Table 5 Test result of UAV Platform.

Method	mAP	FPS	Model Size	
RTS-Net	88.1	21.8	16.6	
RTS-Net (TensorRT FP16)	88.1	40.7	19.8	
RTS-Net (TensorRT INT8)	88.0	54.1	26.4	

UAV platform testing

To verify whether our method can truly meet the real-time inference speed required for UAV operation, we deployed our method on a UAV platform and conducted tests. The UAV platform we used in the experiment uses NVIDIA Jetson Xavier NX as the visual algorithm processor, and the software environment includes PyTorch 1.8.0, JetPack 4.6.1 and OpenCV 4.5.4. The experimental results are shown in Table 5. The results indicate that under the PyTorch deep learning framework, our RTS-Net achieved an inference speed of only 21.8 FPS, which does not meet the requirements for real-time detection. However, after using TensorRT for acceleration, the inference speed of RTS-Net increased to 40.7 FPS, which meets the real-time detection needs of UAV for ground objects. Additionally, the detection accuracy was almost unaffected, and the inference engine size was only 19.8 MB, which does not place a significant burden on UAV storage. Finally, after further quantizing the model parameters, the inference speed increased to 54.1 FPS, with negligible changes in detection accuracy. In summary, the experimental results demonstrate that our method can meet the high-precision real-time detection requirements of UAV for ground objects.

Failure cases

Although our method exhibits excellent detection performance in most scenarios, occasional detection failures still occur in some extreme scenarios. As illustrated in Fig. 10. In Fig. 10a, some of the densely parked motors and bicycles at the top were not completely detected. In Fig. 10c, several cars at the top were partially obscured by trees and overpasses and were also not completely detected. This indicates that our method still has some missed detection issues for small objects in particularly complex backgrounds. In Fig. 10b, a black car in the lower-left corner was not accurately detected, and in Fig. 10d, several people on the stairs in the upper-left corner were also not accurately detected. This shows that our method’s recognition performance decreases in low-light conditions such as at night. In the future, we will continue to work on improving the detection accuracy of UAV in various complex scenarios.Fig. 10 Illustration of failure cases.

Conclusion

In this paper, a novel real-time small object detection network RTS-Net is proposed for urban patrol. Given the problem that small-scale objects with scarce visual information occupy a great proportion of UAV images, we first designed the MFFM module, which enhances the expressive ability of different scale features by adaptive weighted fusion of semantically-rich deep features with detailed information-rich shallow features, thus improving the detection ability of small-scale objects. In order to improve the UAV’s ability to accurately distinguish between object and background in a vast and complex environment, we proposed a CADM module in combination with the attention mechanism, which enables the object detection model to effectively focus on the attention region. Finally, we architected the lightweight RFFM module for reducing the computation of the model and improving the inference speed. In the DRP-Dataset that we constructed, our proposed that RTS-Net achieves 89.9% mAP detection accuracy and 163.9 FPS inference speed, which outperforms all object mainstream object detection methods. Pertinently, for small-scale object classes like person, bicycle, and motor, RTS-Net exhibited superior precision. Furthermore, the robustness experiments show that the proposed RTS-Net can maintain good detection results under adverse conditions, including suboptimal lighting and atmospheric perturbations. The experiments in the VisDrone dataset prove that RTS-Net has strong generalization ability in a variety of different complex scenarios and a variety of different UAV platforms. In the future, we will evaluate the effectiveness of the proposed method in more challenging scenarios to further improve the reliability of UAV patrol.

Author contributions

All authors contributed extensively to this manuscript. Y.W. and J.Z.(Jie Zhang). developed the methodology. Y.W. conceived the experiment(s), J.Z.(Jie Zhang) and J.Z.(Jian Zhou) conducted the experiment(s), Y.W. analysed the results, Y.W. drafted the original manuscript, J.Z.(Jie Zhang) and J.Z.(Jian Zhou) reviewed and edited the manuscript. All authors have read and agreed to the published version of the manuscript.

Data availability

Data is available upon request. Please contact the corresponding author by email to obtain the data used in the study.

Competing interests

The authors declare no competing interest.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Chen P Dang Y Liang R Zhu W He X Real-time object tracking on a drone with multi-inertial sensing data IEEE Trans. Intell. Transp. Syst. 2017 19 131 139 10.1109/TITS.2017.2750091
Chen, P., Dang, Y., Liang, R., Zhu, W. & He, X. Real-time object tracking on a drone with multi-inertial sensing data. IEEE Trans. Intell. Transp. Syst. 19, 131–139 (2017).10.1109/TITS.2017.2750091
2. Huang, C., Chen, P., Yang, X. & Cheng, K.-T. T. Redbee: A visual-inertial drone system for real-time moving object detection. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 1725–1731 (IEEE, 2017).
3. Ye T Dense and small object detection in uav-vision based on a global-local feature enhanced network IEEE Trans. Instrum. Meas. 2022 71 1 13
Ye, T. et al. Dense and small object detection in uav-vision based on a global-local feature enhanced network. IEEE Trans. Instrum. Meas. 71, 1–13 (2022).
4. Khan SD Alarabi L Basalamah S A unified deep learning framework of multi-scale detectors for geo-spatial object detection in high-resolution satellite images Arab. J. Sci. Eng. 2022 47 9489 9504 10.1007/s13369-021-06288-x
Khan, S. D., Alarabi, L. & Basalamah, S. A unified deep learning framework of multi-scale detectors for geo-spatial object detection in high-resolution satellite images. Arab. J. Sci. Eng. 47, 9489–9504 (2022).10.1007/s13369-021-06288-x
5. Yang, C., Huang, Z. & Wang, N. Querydet: Cascaded sparse query for accelerating high-resolution small object detection. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 13668–13677 (2022).
6. Hoanh, N. & Pham, T. V. Focus-attention approach in optimizing detr for object detection from high-resolution images. Knowl.-Based Syst. 296, 111939 (2024).
7. Meethal, A., Granger, E. & Pedersoli, M. Cascaded zoom-in detector for high resolution aerial images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2046–2055 (2023).
8. Liu, J. & Chen, J. A coarse to fine framework for object detection in high resolution image. arXiv preprint (2023). arXiv:2303.01219.
9. Xiaolin F Small object detection in remote sensing images based on super-resolution Pattern Recogn. Lett. 2022 153 107 112 10.1016/j.patrec.2021.11.027
Xiaolin, F. et al. Small object detection in remote sensing images based on super-resolution. Pattern Recogn. Lett. 153, 107–112 (2022).10.1016/j.patrec.2021.11.027
10. Wu J Pan Z Lei B Hu Y Fsanet: Feature-and-spatial-aligned network for tiny object detection in remote sensing images IEEE Trans. Geosci. Remote Sens. 2022 60 1 17 10.1109/TGRS.2022.3228927
Wu, J., Pan, Z., Lei, B. & Hu, Y. Fsanet: Feature-and-spatial-aligned network for tiny object detection in remote sensing images. IEEE Trans. Geosci. Remote Sens. 60, 1–17 (2022).10.1109/TGRS.2022.3228927
11. Yang, X. et al. H2rbox: Horizontal box annotation is all you need for oriented object detection. arXiv preprint (2022). arXiv:2210.06742.
12. Xie X Fewer is more: Efficient object detection in large aerial images Science China Inf. Sci. 2024 67 112106 10.1007/s11432-022-3718-5
Xie, X. et al. Fewer is more: Efficient object detection in large aerial images. Science China Inf. Sci. 67, 112106 (2024).10.1007/s11432-022-3718-5
13. Li, C. et al. Learning object-wise semantic representation for detection in remote sensing imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 20–27 (2019).
14. Fan, X. et al. A small ship object detection method for satellite remote sensing data. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing (2024).
15. Gaszczak, A., Breckon, T. P. & Han, J. Real-time people and vehicle detection from uav imagery. In Intelligent robots and computer vision XXVIII: algorithms and techniques, vol. 7878, 71–83 (SPIE, 2011).
16. Sokalski, J., Breckon, T. P. & Cowling, I. Automatic salient object detection in uav imagery. In Proc. 25th International Conference on Unmanned Air Vehicle Systems, 11–1 (Citeseer, 2010).
17. Cao, X., Wu, C., Yan, P. & Li, X. Linear svm classification using boosting hog features for vehicle detection in low-altitude airborne videos. In 2011 18th IEEE International Conference on Image Processing, 2421–2424 (IEEE, 2011).
18. Moranduzzo, T. & Melgani, F. A sift-svm method for detecting cars in uav images. In 2012 IEEE International Geoscience and Remote Sensing Symposium, 6868–6871 (IEEE, 2012).
19. Moranduzzo T Melgani F Detecting cars in uav images with a catalog-based approach IEEE Trans. Geosci. Remote Sens. 2014 52 6356 6367 10.1109/TGRS.2013.2296351
Moranduzzo, T. & Melgani, F. Detecting cars in uav images with a catalog-based approach. IEEE Trans. Geosci. Remote Sens. 52, 6356–6367 (2014).10.1109/TGRS.2013.2296351
20. Su A Sun X Liu H Zhang X Yu Q Online cascaded boosting with histogram of orient gradient features for car detection from unmanned aerial vehicle images J. Appl. Remote Sens. 2015 9 096063 096063 10.1117/1.JRS.9.096063
Su, A., Sun, X., Liu, H., Zhang, X. & Yu, Q. Online cascaded boosting with histogram of orient gradient features for car detection from unmanned aerial vehicle images. J. Appl. Remote Sens. 9, 096063–096063 (2015).10.1117/1.JRS.9.096063
21. Moranduzzo T Melgani F Bazi Y Alajlan N A fast object detector based on high-order gradients and gaussian process regression for uav images Int. J. Remote Sens. 2015 36 2713 2733 10.1080/01431161.2015.1043760
Moranduzzo, T., Melgani, F., Bazi, Y. & Alajlan, N. A fast object detector based on high-order gradients and gaussian process regression for uav images. Int. J. Remote Sens. 36, 2713–2733 (2015).10.1080/01431161.2015.1043760
22. Scaramuzza D Vision-controlled micro flying robots: from system design to autonomous navigation and mapping in gps-denied environments IEEE Robotics Autom. Mag. 2014 21 26 40 10.1109/MRA.2014.2322295
Scaramuzza, D. et al. Vision-controlled micro flying robots: from system design to autonomous navigation and mapping in gps-denied environments. IEEE Robotics Autom. Mag. 21, 26–40 (2014).10.1109/MRA.2014.2322295
23. Meier, L. et al. Pixhawk: A micro aerial vehicle design for autonomous flight using onboard computer vision. Auton. Robot. 33, 21–39 (2012). Meier, L. et al. Pixhawk: A micro aerial vehicle design for autonomous flight using onboard computer vision. Auton. Robot. 33, 21–39 (2012).
24. Xu Y Yu G Wang Y Wu X Ma Y A hybrid vehicle detection method based on viola-jones and hog+ svm from uav images Sensors 2016 16 1325 10.3390/s16081325 27548179
Xu, Y., Yu, G., Wang, Y., Wu, X. & Ma, Y. A hybrid vehicle detection method based on viola-jones and hog+ svm from uav images. Sensors 16, 1325 (2016).27548179 10.3390/s16081325
25. Chen X Xiang S Liu C-L Pan C-H Vehicle detection in satellite images by hybrid deep convolutional neural networks IEEE Geosci. Remote Sens. Lett. 2014 11 1797 1801 10.1109/LGRS.2014.2309695
Chen, X., Xiang, S., Liu, C.-L. & Pan, C.-H. Vehicle detection in satellite images by hybrid deep convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 11, 1797–1801 (2014).10.1109/LGRS.2014.2309695
26. Bejiga, M. B., Zeggada, A. & Melgani, F. Convolutional neural networks for near real-time object detection from uav imagery in avalanche search and rescue operations. In 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 693–696 (IEEE, 2016).
27. Ammour N Deep learning approach for car detection in uav imagery Remote Sens. 2017 9 312 10.3390/rs9040312
Ammour, N. et al. Deep learning approach for car detection in uav imagery. Remote Sens. 9, 312 (2017).10.3390/rs9040312
28. Bazi Y Melgani F Convolutional svm networks for object detection in uav imagery IEEE Trans. Geosci. Remote Sens. 2018 56 3107 3118 10.1109/TGRS.2018.2790926
Bazi, Y. & Melgani, F. Convolutional svm networks for object detection in uav imagery. IEEE Trans. Geosci. Remote Sens. 56, 3107–3118 (2018).10.1109/TGRS.2018.2790926
29. ultralytics. ultralytics github repository (2023).
30. Ren, S., He, K., Girshick, R. & Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems 28 (2015).
31. Dai, J., Li, Y., He, K. & Sun, J. R-fcn: Object detection via region-based fully convolutional networks. Advances in Neural Information Processing Systems 29 (2016).
32. Liu, W. et al. Ssd: Single shot multibox detector. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, 21–37 (Springer, 2016).
33. Redmon, J., Divvala, S., Girshick, R. & Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 779–788 (2016).
34. Redmon, J. & Farhadi, A. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7263–7271 (2017).
35. Redmon, J. & Farhadi, A. Yolov3: An incremental improvement. arXiv preprint (2018). arXiv:1804.02767.
36. Bochkovskiy, A., Wang, C. -Y. & Liao, H. -Y. M. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint (2020). arXiv:2004.10934.
37. Ge, Z., Liu, S., Wang, F., Li, Z. & Sun, J. Yolox: Exceeding yolo series in 2021. arXiv preprint[SPACE]arXiv:2107.08430 (2021).
38. Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint (2020). arXiv:2010.11929.
39. Chen, Q. et al. You only look one-level feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13039–13048 (2021).
40. Vaddi, S. Efficient object detection model for real-time UAV applications. Ph.D. thesis, Iowa State University (2019).
41. Zhang, P., Zhong, Y. & Li, X. Slimyolov3: Narrower, faster and better for real-time uav applications. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
42. Zhu, X., Lyu, S., Wang, X. & Zhao, Q. Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured scenarios. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2778–2788 (2021).
43. Cao, S., Liu, Y., Lasang, P. & Shen, S. Detecting the objects on the road using modular lightweight network. arXiv preprint (2018). arXiv:1811.06641.
44. Ye T Real-time object detection network in uav-vision based on cnn and transformer IEEE Trans. Instrum. Meas. 2023 72 1 13 37323850
Ye, T. et al. Real-time object detection network in uav-vision based on cnn and transformer. IEEE Trans. Instrum. Meas. 72, 1–13 (2023).37323850
45. Du, B., Huang, Y., Chen, J. & Huang, D. Adaptive sparse convolutional networks with global context enhancement for faster object detection on drone images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13435–13444 (2023).
46. Lin, T.-Y. et al. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2117–2125 (2017).
47. Liu, S., Qi, L., Qin, H., Shi, J. & Jia, J. Path aggregation network for instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8759–8768 (2018).
48. Ghiasi, G., Lin, T.-Y. & Le, Q. V. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7036–7045 (2019).
49. Kim, S. -W., Kook, H. -K., Sun, J. -Y., Kang, M. -C. & Ko, S. -J. Parallel feature pyramid network for object detection. In Proceedings of the European Conference on Computer Vision (ECCV), 234–250 (2018).
50. Zhao Q M2det: A single-shot object detector based on multi-level feature pyramid network Proc. AAAI Conf. Artif. Intell. 2019 33 9259 9266
Zhao, Q. et al. M2det: A single-shot object detector based on multi-level feature pyramid network. Proc. AAAI Conf. Artif. Intell. 33, 9259–9266 (2019).
51. Tan, M., Pang, R. & Le, Q. V. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10781–10790 (2020).
52. Hou, Q., Zhou, D. & Feng, J. Coordinate attention for efficient mobile network design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13713–13722 (2021).
53. Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7132–7141 (2018).
54. Woo, S., Park, J., Lee, J.-Y. & Kweon, I. S. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), 3–19 (2018).
55. Du, D. et al. Visdrone-det2019: The vision meets drone object detection in image challenge results. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
56. Cao, Y. et al. Visdrone-det2021: The vision meets drone object detection challenge results. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2847–2854 (2021).
57. Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10012–10022 (2021).
58. Carion, N. et al. End-to-end object detection with transformers. In European Conference on Computer Vision, 213–229 (Springer, 2020).
59. Zhang, S. et al. Dense distinct query for end-to-end object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7329–7338 (2023).
60. Zhang, H. et al. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint (2022). arXiv:2203.03605.
61. Tan, M. & Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, 6105–6114 (PMLR, 2019).
62. Chen, K. et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint (2019). arXiv:1906.07155.
