
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

71394
10.1038/s41598-024-71394-3
Article
Fine classification of rice fields in high-resolution remote sensing images
Zhao Lingyuan
Luo Zifei
Zhou Kuang
Yang Bo
Zhang Yan zyan0htzh@gmail.com

Technology Research and Development Center, Huantian Wisdom Technology, Meishan, 620564 China
6 9 2024
6 9 2024
2024
14 207781 2 2024
27 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Fine-grained management of rice fields can enhance the yield and quality of rice crops. Challenges in achieving fine classification include interference from similar vegetation, the irregularity of natural field shapes, and complex scale variations. This paper introduces Rice Attention Cascade Network (RACNet), for the fine classification of rice fields in high-resolution satellite remote sensing imagery. The network employs the Hybrid Task Cascade network as the base framework and uses spectral and indices mixed multimodal data as input to reinforce the feature differentiation of similar vegetation. Initially, a Channel Attention Deformable-ResNet (CAD-ResNet) was designed to enhance the feature representation of rice on different channels. Deformable convolution improves the ability of CAD-ResNet to capture irregular field shapes. Then, to address the issue of complex scale changes, the multi-scale features extracted by the CAD-ResNet are progressively fused using an Asymptotic Feature Pyramid, reducing the loss of scale information between non-adjacent layers. Experiments on the Meishan rice dataset show that the proposed method is capable of accurate instance segmentation for fragmented or irregularly shaped rice fields. The evaluation metric AP50 of RACNet reaches 50.8%.

Keywords

Crop classification
High-resolution satellite remote sensing imagery
Hybrid Task Cascade network
Subject terms

Computer science
Environmental sciences
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Rice tops the list of major food crops globally in terms of planted area and production, especially in Asia1,2. As a result, the high yield and quality of rice is of paramount importance. Fine-tuned management practices that monitor each rice field plot individually can improve soil quality, monitor diseases, and enable proper fertilization to ensure long-term soil sustainability, leading to higher rice yields and quality3.

Remote sensing in agriculture is a technology that utilizes remote sensing satellites, unmanned aerial vehicles4, multispectral sensors and other remote sensing tools to monitor and manage agricultural activities5. Remote sensing technology has great potential for application in rice monitoring. Specifically, satellite sensors can provide spectral information, high-resolution images, and other multimodal and multidimensional data to provide comprehensive data for rice detection. In areas with discontinuous altitudes, the terrain fluctuates significantly, with interlaced topographic features such as mountains and hills coexisting. Discontinuous altitudes make it difficult to reclaim large contiguous areas of farmland. Cropland may be divided into relatively small and scattered farmlands, leading to fragmented planting. Currently, few research tasks focus on the fine classification of crops based on satellite images in areas with discontinuous altitudes, including the classification and extraction of major food crops. In fact, satellite-based crop monitoring is of great importance.

Our main work is to realize the fine classification of rice fields in southwestern China using images captured by remote sensing satellites, with a spatial resolution of 0.5–2 m. To address the challenges associated with fine rice classification, we must overcome the following obstacles.RGB images mostly preserve the color features of plants, but the image resolution limits the expression of morphological features of rice. The presence of shrubs or plants with similar colors can result in misclassification.

In planning areas with discontinuous elevation, or non-standard farmland, the fragmentation and irregularity of rice cultivation bring challenges to the fine classification of rice in remote sensing images.

Remote sensing images encompass diverse scale features, with the characteristics of the target area undergoing significant changes at different scales.

The following strategies are used to solve these problems.

We used vegetation indices to process the raw images (R, G, B, NIR) instead of simply using RGB images as inputs to the network. Multimodal data containing visible images and vegetation index images jointly characterize rice features. The multi-modal data focuses on the feature contributions of different channels of remote sensing images to strengthen the categories differences.

To cope with the fragmentation of rice cultivation and the irregularity of natural field shapes, Deformable Convolution (DCN) is applied to the feature extraction stage. The learning of offsets allows the convolution kernel to be adjusted to the focus area. The ability of the feature extraction layer to capture the contours of a single field type will be improved.

The main component of the field is bare soil or hardened roads. Its NDVI (Normalized Difference Vegetation Index) index is much lower than that of rice. In this paper, the ECA (Efficient Channel Attention) mechanism strengthens the channel attention to learn the weight relationship between RGB and NDVI channels when extracting features. This mechanism enhances the sensitivity to rice and reduces the misclassification of dense targets. At the same time, the network suppresses the weights of non-focused regions to optimize the segmentation outlines.

In order to gain detailed scale information of remote sensing images, we use AFPN (Asymptotic Feature Pyramid Network) for feature fusion. Feature fusion of non-adjacent layers preserves the differences in scale and reduces the feature loss in the fusion stage.

Related works

Multimodal data

Multimodal Remote Sensing Image Interpretation (MRSII) is an emerging direction in the field of Earth observation and computer vision. It has greater application than single modal. Multimodal data refer to information in multiple data types or modalities from different sensors or data sources. In crop classification tasks, multimodal data can be derived from visible images, infrared images, hyperspectral data, radar data, etc. These abundant data sources provide multifaceted information about soil, vegetation, moisture, and crop conditions6,7.

A common multi-modal approach is to combine visible images and hyperspectral data. Visible images offer morphological and color information about the crop, while hyperspectral data captures spectral features of the vegetation. Researchers have applied mixed-modal features to classification algorithms such as Support Vector Machines (SVM), XGBoost algorithm and deep learning methods for effective crop classification8,9.

Instance segmentation network for crop classification

In the field of agricultural remote sensing, numerous excellent traditional methods have been developed to address crop classification challenges. The imagery data of farmland, vegetation and soil captured by agricultural remote sensing technologies is usually large-scale and multidimensional, requiring highly sophisticated data processing and analysis. Machine learning technology, by learning from data and automatically improving performance, serves as a powerful tool in agricultural remote sensing. In this domain, machine learning technology is applied to data classification, pattern recognition, prediction, and decision support. Early methods for extracting cultivation information from remote sensing data commonly included machine learning-based approaches such as random forests10, decision tree classification11, and maximum likelihood methods12. In conjunction with machine learning, some outstanding techniques are also employed for crop classification, such as multi-temporal sequence analysis13. Temporal sequence-based classification methods utilize data from different time points or time sequences to observe and analyze changes in crops over time14. The combination of time series and high-resolution satellite imagery can better reflect crop phenological characteristics. However, high-resolution satellites often have narrow widths and longer revisit periods, making it challenging to obtain high-quality, high-resolution images for long time series.

Instance segmentation methods show superior performance in crop classification tasks using single-date high-resolution satellite remote sensing images. These methods provide distinct labels for different object instances that belong to the same object class. Instance segmentation methods can be categorized into two types: single-stage detectors and two-stage detectors.

Single-stage instance segmentation, inspired by single-stage object detection, aims to directly differentiate between different objects without additional proposal boxes. The YOLACT15 network is a simple and efficient example, utilizing the fusion of prototype masks and detection boxes to generate predicted masks. While it achieves improved speed, its accuracy in instance segmentation is lower compared to two-stage methods. In a lychee tree canopy classification task16, the AP(Average Precision) of the YOLACT network reached 96.26%. Two-stage instance segmentation comprises two approaches. Firstly, the bottom-up approach treats each instance as a category, embedding each pixel into a space representing the category and subsequently separating different instances through clustering. Secondly, the top-down approach, also known as detect-then-segment, first locates each instance through object detection, then performs semantic segmentation within each detected box. Representative models of this approach include Mask-RCNN17, PANet18, Mask Scoring R-CNN19, etc.

Mask R-CNN is a typical top-down instance segmentation algorithm. It extends the object classification and regression branches by adding a semantic segmentation branch for predicting each region of interest (ROI). However, Mask R-CNN relies on the results of object detection. In the context of the center pivot irrigation systems detection task20, Mask R-CNN achieved an AP of 80.21, but due to its weak small object detection capability, the APs were only 16.91%. The use of a deeper ResNeXt101-FPN backbone network effectively enhances segmentation accuracy21.

The Hybrid Task Cascade (HTC) instance segmentation method is an improvement based on the Mask R-CNN model. HTC introduces a cascaded structure and a fully convolutional network into the network22. This approach exhibits excellent detection and segmentation performance, making it suitable for classification tasks with high-resolution satellite remote sensing images23.

Channel attention mechanism

The core concept of attention mechanisms lies in selectively highlighting key information in input data while suppressing less relevant details, thereby directing focus towards the most crucial data. In the visual domain, various dimensions of attention mechanisms include channel attention, spatial attention, temporal attention, and branch attention.

Channel attention is designed to emphasize channels critical for the current task by weighting features from different channels. The Squeeze and Excitation Network (SENet) offers a classic channel attention solution24. SENet employs a squeeze-and-excitation approach to establish relationships between feature map channels, allowing adaptive adjustment of the weights of feature map channels. The ECA (Efficient Channel Attention) attention mechanism is another form of channel attention25. Due to the dimensionality reduction impact on data caused by the fully connected layer in the SE module, the ECA network employs adaptive convolution kernels and one-dimensional convolution to mitigate this influence26.

Feature fusion

Feature pyramid is a common feature fusion strategy in instance segmentation, often employed to address scale variation issues27. FPN (Feature Pyramid Network) is the most widely used feature pyramid structure28. The FPN module integrates contextual information (high-level semantic information) from the top-down model, increasing the resolution of the feature maps29. However, multiscale feature fusion typically occurs only between adjacent levels of feature layers. When there is a significant semantic gap between non-adjacent high-level and low-level feature layers, especially between the topmost and bottommost layers, FPN tends to lose some details during feature fusion. The structure of FPN is illustrated in Fig. 1.Fig. 1 The Feature Pyramid Network (FPN) fuses the feature maps from the higher layers (high resolution, but weak semantic strength) with those from the lower layers (low resolution, but strong semantic strength), thereby generating feature maps with rich semantic information at each scale. In the top-down pathway, the Feature Pyramid Network focuses only on information from adjacent scales.

Based on FPN, PAFPN (Path Augmentation Feature Pyramid Network) introduces a bottom-up pathway to the feature pyramid network. PAFPN addresses the lack of detailed low-level feature information within the high-level features of FPN. However, detailed information from low-level features may be lost or degraded during propagation and interaction processes30. To overcome the limitations, Asymptotic Feature Pyramid Network (AFPN) is introduced31. During the bottom-up feature extraction process, AFPN initiates the fusion by combining two low-level features with different resolutions. It progressively incorporates high-level features into the fusion process, eventually obtaining the top-level feature. This fusion approach helps mitigate significant semantic gaps between non-adjacent levels.

Materials

Study area

Meishan belongs to Sichuan Province, China. Located on the southwestern edge of the Sichuan Basin, Meishan exhibits significant topographical variations. The elevation in Meishan varies due to the diverse terrain, generally ranging between 500 and 3000 m above sea level. The topographic map of Meishan is shown in Fig. 2.Fig. 2 Topographic map of Meishan and the study area. The data source for Fig. 2 is a public image available at https://earthengine.google.com/. The processing software used for creating the figure includes Microsoft Visio and ArcGIS Pro (version 3.3.0).

Meishan experiences a subtropical monsoon climate, characterized by warm and rainy summers and relatively dry winters. The annual temperatures conducive to agricultural production. Common cereal crops in the region include rice, wheat, and corn.

However, due to the undulating terrain, intricate topography and numerous river valleys in Meishan, it is difficult to realize continuous planting on a large scale. The agricultural land is prone to fragmentation, resulting in small and scattered plots.

Satellite data

The imagery data for this paper was captured by the JL-1 satellite. JL-1 is a high-resolution remote sensing satellite independently developed by Jilin Province in China. It captures imagery with a swath width exceeding 150 km, revisiting the same area every day. The panchromatic resolution is 0.5 m, and the multispectral resolution is 2 m. JL-1 satellite is primarily employed for resource surveys and monitoring in fields such as agriculture, forestry, water resources, and urban planning. The spectral channel information is presented in Table 1 below.Table 1 The spectral bands information of JL-1 satellite.

Wave band	wavelength range(nm)	
PAN	450–800	
Blue(B1)	450–510	
Green(B2)	510–580	
Red(B3)	630–690	
Near-Infrared(B4)	770–895	

We select high-resolution images taken during the flowering period of rice in July as the original data. The original L1 data undergoes radiometric calibration, orthorectification, image registration, and image fusion. We use ArcGIS Pro as our annotation tool. The mask raster is exported for preprocessing. In the preprocessing stage, we cut the satellite images into a size of 512*512 and set a 50% chance to randomly flip the images to achieve data augmentation. After completing the image preprocessing, we create the COCO format dataset. And the training set and validation set are divided at a ratio of 4:1. There are a total of 4052 images in the training set, and the validation set contains 1012 images.

Methodology

The technical route used in this paper to realize the rice fine classification consists of three parts: satellite image preprocessing; dataset preparation; and rice classification based on instance segmentation. The detailed explanation of the above content will be presented in this section.

Multimodal data fusion

In rice classification tasks, index features can serve as informative descriptors to alleviate the “same spectrum for different substances” issue. Vegetation index is a radiometric value that reflects the relative abundance and activity of green living vegetation. It is often used to characterize the physiological status of vegetation, green biomass and vegetation productivity in the study area. NDVI (Normalized Difference Vegetation Index) provides robust vegetation growth features by calculating the normalized difference in reflectance between visible and near-infrared spectral bands. Its inherent vegetation dynamic information, such as the greenness and leaf coverage during the mid-growth stage, makes it a powerful indicator for distinguishing rice from other land features. The NDVI calculation formula is as follows.1 NDVI=(NIR-Red)/(NIR+Red)

where Red and NIR correspond to the reflectance values in the red and near-infrared spectral bands, respectively. Simultaneously, visible light bands are fused with NDVI, combining color features from the same scene with index features, participating together in training and decision-making processes. The Joint Representation of R, G, B, and NDVI maps information from multiple modalities into a unified multimodal vector space. Here, R, G, B represent inputs from visible light, and NDVI represents the index input. xvis represents input from visible light, and xNDVI represents the NDVI index input. Jointly fusing multiple input modalities xvis,xNDVI yields a multimodal representation xm=fxvis,xNDVI , xm participate in the classification task.

Fine rice classification cascade network

We improve the end-to-end instance segmentation model, RACNet (Rice Attention Cascade Network), for fine classification of rice fields tasks based on the Hybrid Task Cascade (HTC) framework. RACNet consists of four components: (1) a CAD-Resnet backbone network, enhanced based on resnet50 for feature extraction; (2) an asymptotic feature extraction module for achieving multiscale feature fusion in remote sensing; (3) the cascade structure for concurrently performing bounding box regression and mask prediction; (4) a semantic segmentation fusion branch for acquiring additional contextual information. The network architecture of RACNet is illustrated in Fig. 3. Details of components (1) and (2) will be elaborated in Sections "Deformable convolution for addressing complex field shapes" and "Channel attention for enhancing category differentiation".Fig. 3 The network architecture of RACNet.

Input: We use the multimodal joint representation xm introduced in Section "Multimodal data fusion" as the input for the instance segmentation network. The size of the input image is adjusted to x∈RH∗W∗C with a spatial resolution H×W of 512 × 512 and a channel number C of 4. The adjusted image serves as the input for the first layer of the CAD-ResNet backbone network.

CAD-ResNet Backbone Network: In the down-sampling stage, we employ ResNet-50 as the base network for feature extraction. Deformable convolutions serve as the basic convolution modules for the last two stages of the backbone network. Simultaneously, ECA attention is added at each layer of the feature extraction network.

Asymptotic Feature Extraction Pyramid: We construct the corresponding top-down network AFPN, consisting of asymptotic feature fusion branches. Each branch progressively fuses and aligns features upwards, incorporating features at different scales. Adaptive spatial fusion has also been added.

Hybrid Task Cascade: This structure comprises the RPN region proposal generation module, ROI pooling, bounding box regression, and mask prediction modules. The bounding box regression and mask prediction tasks are interleaved. Information pathways are added between mask branches. Formula (2)(3) represent this process of cascade interleaving.2 xtbox=Px,rt-1,rt=Htxtbox,

3 xtmask=Px,rt,mt=Mtxtmask.

x represents the CNN features of the backbone network, xtbox denotes the detection boxes, xtmask represents mask features, P is the set operator. Ht and Mt represent the detection and segmentation heads. rt , mt respectively indicate the prediction results for region proposal boxes and masks in the current stage t.

For stage t, the mask prediction features come from the mask features of the t stage and the mask prediction results from the t-1 stage. The formula is as follows:4 xtmask=Px,rt,mt=MtFxtmask,mt-1-,

5 Fxtmask,mt-1=xtmask+Gtmt-1-.

Semantic Segmentation Fusion Branch: The network adds a fully convolutional branch for pixel-level segmentation. The branch S(x) is fused with regression boxes and mask branches to acquire more contextual information. The fusion method is shown in formula (6).6 xtmask=Px,rt+PS(x),rt

Output: The rt and mt from the final stage are the ultimate predicted bounding boxes and masks.

Deformable convolution for addressing complex field shapes

The varied terrain in Meishan results in numerous irregular and fragmented rice paddies. Traditional fixed-size convolutional receptive fields struggle to sensitively capture the overall shape and boundaries of these land parcels. This limitation makes the original ResNet50 feature extraction network less effective in accurately understanding the complexity of the terrain.

Deformable convolutions overcome the limitations of fixed-size convolution methods, allowing flexible capture of land parcel contour information as shown in Fig. 4.Fig. 4 (a) illustrates the process of standard convolution for extracting rice features, (b) depicts the process of deformable convolution for extracting rice features, with a convolution kernel size of 3. The high-resolution remote sensing imagery used in the study is sourced from Huantian Wisdom Technology Co.,Ltd.

During feature extraction, convolution units sample features from the input feature map at fixed positions, as shown in Fig. 4a. However, different positions may correspond to rice field information of varying scales or deformations. Moreover, the effectiveness of object detection relies heavily on feature-based bounding boxes. Therefore, these layers need methods to automatically adjust scale or receptive field. The fixed convolution is not the optimal approach, especially for non-grid-like rice fields. We employ deformable convolutions to replace the original fixed convolutions, enhancing the learning capability for irregular or significantly rotated field shapes, as illustrated in Fig. 4b.

Compared with standard convolution, deformation convolution samples the pixel values based on the convolved offset coordinates and performs bilinear interpolation to generate a new feature map, as shown in Fig. 5. In the x feature layer, xp represents the feature value at position p, w(p) represents the weight at position p, p0 denotes the central point’s position, pn represents the relative positions of surrounding points, and R represents all relative positions of the convolution kernel. The formula (7) indicates that for the convolution at position p0, the sum of convolutions for this point and its surrounding 8 points p0+pn needs to be calculated to obtain the element value yp0 at position p0 on the output feature map.7 yp0=∑pn∈Rwpn·xp0+pn

Fig. 5 (a) represents a regular convolution operation. (b) and (c) illustrate deformable convolutions.

In deformable convolution, a positional offset Δpn, which includes the coordinates x and y, is added to the calculation as shown in formula (8).8 yp0=∑pn∈Rwpn·xp0+pn+Δpn

In this paper, a bilinear interpolation method is used to compute the eigenvalues of the output of the convolutional network.

Channel attention for enhancing category differentiation

Enhanced Channel Attention (ECA) is employed in this study to address the challenges caused by differences in land types and interference from similar vegetation. The joint representation xm=fxvis,xNDVI, consisting of visible light bands and NDVI index images, is the input for the instance segmentation network. Field ridges are typically composed of bare soil or hardened roads, they have small differences in reflection between the visible and near-infrared bands, leading to NDVI values close to zero or negative. For healthy vegetation, the reflection value in the green channel (G) is usually higher than that in the red (R) and blue (B) channels, with an NDVI index greater than 0.3. Moreover, different vegetation types exhibit variations in NDVI channel values.

To address these differences in land types and vegetation characteristics, the ECA attention mechanism is specifically applied to learn the weight relationships between the red, green, blue channels, and NDVI channel. In this way ECA reinforces the differences in channels for the various land cover types. ECA achieves this by performing a fast one-dimensional convolution of size k to generate channel weights, where the value of k is adaptively determined as a function of the channel dimension C as shown in Fig. 6.Fig. 6 Enhanced Channel Attention (ECA) module.

The input feature map undergoes global average pooling (GAP), transforming the feature map into a vector of size [1, 1, C]. The adaptive one-dimensional convolution kernel size is calculated based on the channel number C, and the adaptive function is represented by the formula (9).9 k=log2(C)a+ba,

10 ω=SigmoidC1Dky.

where a = 1, b = 2, employing the sigmoid activation function yields the weights for each channel of the feature map.

After global average pooling, the ECA attention mechanism does not reduce the dimensionality of channels. It considers each channel and its k neighbors for local cross-channel interaction information. In the instance segmentation task, the ECA attention mechanism helps to reduce the feature loss of discontinuous paddy targets on the image and minimize the generation of false proposal boxes for similar vegetation. By emphasizing key channels and suppressing non-concerned regions, this mechanism helps to extract the boundaries and structures of rice regions more accurately. It enables the network to better adapt to remote sensing images of different vegetation types.

Rice feature fusion strategy

In addition to spectral, shape, and texture features, scale features in remote sensing images are of significant importance. In the target detection phase, the fusion of multi-scale features enhances the network’s robustness to scale variations. The AFPN reduces feature loss on non-adjacent scales through the combination of residual connections and adaptive weight adjustment, resulting in a more accurate feature representation. The network extracts final layer features from each feature layer of the backbone at different scales denoted as {C2, C3, C4, C5}. Specifically, the progressive feature fusion strategy is summarized as follows:1. The neck extracts the final layer features from each feature layer of the Backbone, forming {C2, C3, C4, C5};

2. Initially, the AFPN module inputs Low-Level features C2 and C3, performs ASF_s (Adaptively Spatial Fusion) fusion to obtain multi-scale features {Cs:2, Cs:3}, where s = 2 represents the second stage.

3. The AFPN module incorporate the next Level feature layer C4, forming {Cs-1:2, Cs-1:3, C4}, and perform ASF_s fusion to obtain multi-scale features {Cs:2, Cs:3, Cs:4}, where s = 3 represents the third stage.

4. The AFPN module incorporate the High-Level feature layer C5, forming {Cs-1:2, Cs-1:3, Cs-1:4, C5}, and perform ASF_s fusion to obtain multi-scale features {P2, P3, P4, P5}, where s = 4 represents the fourth stage.

Each stage includes an Adaptive Spatial Fusion (ASF) module. The core idea of ASF is to use normalized weights to perform a weighted sum of features from L scales. Specifically, let xijn→l represent the feature vector at position (i, j) from Level N to Level L. The resulting feature vector, denoted as yijl, is obtained through adaptive spatial fusion, a linear combination of multi-level feature vectors xij1→l, xij2→l, xij3→l, and is formulated as follows.11 yijl=αijl·xij1→l+βijl·xij2→l+γijl·xij3→l

where αijl, βijl, γijl are the spatial weights of three features located at Level L, subject to the constraint αijl+βijl+γijl=1. Considering the complex scale variations of rice in remote sensing images, ASF assigns different spatial weights to features extracted at different scales by the backbone network, enhances the importance of key feature layers.

Experiments

Experimental data and evaluation metrics

Our experimental data is sourced from the JL-1 satellite, capturing the region of Meishan in Sichuan Province. We select high-resolution images from July as the original data, and pre-processe them by cutting them into 512*512-sized four-channel images. The annotation information comes from natural field shape masks, divided into two classes: rice and background. The training set and validation set are split in a ratio of 4:1.

In the evaluation stage, Average Precision (AP) is used to evaluate the detection performance of the model for specific categories. AP comprehensively considers the precision (Precision) and recall rate (Recall) of the model under different confidence thresholds. It reflects the accuracy and completeness of the model when detecting objects of a certain category. Precision represents the proportion of true positive examples among the samples predicted as positive examples by the model. Recall rate represents the proportion of true positive examples that are correctly predicted as positive examples by the model among all true positive examples. APsmall, APmedium, and APlarge are typically employed to evaluate instance segmentation models on targets of small, medium, and large sizes, respectively, within specific size intervals.

Loss function

The loss function used in the experiment is the Cross-Entropy loss with label smoothing. In Cross-Entropy loss with label smoothing, the labels are smoothed by adding a small value to the true label and subtracting the same value from all other labels. This helps reduce the model’s overconfidence by encouraging it to produce more uncertain predictions. The formula is given by:12 L(y,y^)=∑c=1C(1-ε)yclogy^c+εClogy^

where y is the true label, y^ is the predicted label, C is the number of classes, and ε=0.1 is the smoothing value.

Comparative experiments

We set the batch size to 8, the number of epochs to 160, and the learning rate to 0.001. We utilize AdamW as the optimizer. All experiments are conducted using four Tesla T4 GPUs, each with 16 GB of memory. The operating system is Ubuntu 22.04.2, with CUDA version 11.5, and Python version 3.10. The main experiments are performed on the Meishan rice dataset, and the improved RACNet is compared with classical instance segmentation networks, including Mask R-CNN, SOLOv2, YOLACT, and YOLACT in comparative experiments. Table 2 illustrates the evaluation metrics for each model. The results for RACNet in the table represent the mean values from six experiments. Over these experiments, the minimum AP50 is 50.3, with a maximum of 51.6. The minimum APsmall is 12.1, with a maximum of 12.5. The minimum APmedium is 43.1, with a maximum of 44.1. The minimum APlarge is 45.6, with a maximum of 46.8.Table 2 The evaluation metrics for each model.

Model	AP50 (%)		Segmentation AP(%)		Train time (h)	
Small	Medium	Large	
Mask R-CNN	46.5	10.9	35.1	36.1	6.3	
SOLOv2	40.5	5.1	16.1	33.9	9.5	
YOLACT	32.5	2.1	22.1	20.9	3.1	
CondInst	43.7	7.7	35.0	31.0	11.2	
RACNet	50.8	12.3	43.7	45.9	22.8	
Significant values are in bold.

From the Table 2, it is evident that Mask R-CNN, a classic two-stage instance segmentation model, exhibits excellent performance, achieving high detection and segmentation accuracy. The AP metrics of Mask R-CNN across various object sizes are only below those of RACNet. SOLOv2 shows lower detection performance compared to other models but performs well on large objects. YOLACT exhibits relatively lower performance, particularly in object segmentation. CondInst demonstrates balanced performance, excelling in both overall detection performance and small object detection. Our designed RACNet achieves the best detection performance, particularly with outstanding performance improvements on medium and large objects. Ultimately, considering overall detection performance, object segmentation performance, and adaptability to different-sized objects, RACNet is the top-performing model in comparative experiments.

In order to compare and analyze the segmentation results of the models intuitively, we visualize the experimental results in Fig. 7.Fig. 7 (a–e) show the test results for five typical scenarios. The first column in the Figure represents the RGB original image, the second column shows the ground truth labels, and the third to seventh columns display the visualization results of Mask R-CNN, SOLO, YOLACT, CondInst, and our proposed RACNet network, respectively.

Combining the Table 2 and the Fig. 7, it can be observed that the two-stage instance segmentation models, Mask R-CNN and RACNet achieve higher accuracy. At the same confidence, two-stage instance segmentation models have fewer missed detections, although it may lose some edge information. As shown in Fig. 7e, Mask R-CNN exhibits issues in predicting field ridges, combining multiple rice paddies into a single instance. SOLOv2, YOLACT, and CondInst all suffer from significant missed detection problems, especially YOLACT, which performs poorly on large objects, as shown in Fig. 7a. The proposed RACNet performs well in detecting and segmenting rice paddies while preserving ridge information, with some minor loss in small objects.

Ablation study

This section presents the ablation experiments conducted on RACNet to observe the impact of different configurations on the experimental results. All models are trained with the same hyperparameters and settings to ensure fairness.

Ablation study on ECA attention mechanism

Our first ablation experiment analyzes the impact of the ECA attention. The table’s first row shows the segmentation performance of the original HTC. The effect of the ECA attention is shown in the Table 3. The inclusion of the ECA attention module in the backbone leads to improvements in both AP50 (accuracy for small objects) and Segmentation AP across small, medium, and large object categories.Table 3 Results of ablation study on ECA attention mechanism.

Model	AP50 (%)		Segmentation AP (%)		
Small	Medium	Large	
HTC	44.8	9.3	36.6	42.7	
HTC + ECA	48.7	11.1	39.7	45.7	
Significant values are in bold.

Specifically, the addition of ECA enhances the model’s ability to accurately detect and segment objects, resulting in increased performance metrics. We visualize the results of the DCN convolution ablation experiment. The visualization results are shown in Fig. 8.Fig. 8 (a–c) show the test results for three typical scenarios. The first column shows the original RGB image, the second column shows the ground truth labels, the third and fourth columns show the visualization results of the original HTC network and the network adding the ECA attention mechanism, respectively.

From the Fig. 8, it can be seen that ECA effectively enhances the prediction of small targets, reducing the number of missed small targets boxes. However, the network with the added ECA mechanism exhibits a slight over-segmentation issue, leading to false positive (FP) targets in the segmentation results.

Ablation study on DCN

The second ablation experiment analyzes the effect of the DCN module. The table’s first row shows the performance of the original HTC. The impact of the DCN module is shown in the Table 4.Table 4 Results of ablation study on DCN.

Model	AP50		Segmentation AP (%)		
Small	Medium	Large	
HTC	44.8	9.3	36.6	42.7	
HTC + DCN	48.3	10.5	39.9	44.6	
HTC + ECA + DCN(CAD-ResNet)	48.9	11.4	40.5	45.1	
Significant values are in bold.

When the DCN is used to replace traditional convolutions in the backbone model, there is an increase of 3.5% in AP50 and a 3.3% increase in APmedium. The other metrics also exhibit slight enhancements. In this ablation experiment, the CAD-ResNet backbone network, augmented with ECA and DCN, achieves the highest scores. We visualized the results of the DCN convolution ablation experiment for intuitive comparison as shown in Fig. 9.Fig. 9 (a–c) show the test results for three typical scenarios. The first column shows the original RGB image, the second column shows the ground truth labels, and the third and fourth columns show the visualization results of the original HTC network and the network using a DCN instead of traditional convolution. The fifth column shows the experimental results using CAD-ResNet as the feature extraction network.

From the Fig. 9, it can be seen that DCN enhances the model’s sensitivity to natural field features, as shown in (b). The segmentation results for irregular-shaped rice paddies have significantly improved. We additionally added the ECA attention mechanism for the ablation experiment, as shown in (a) and (c), indicating that the addition of ECA suppresses paddy field information.

Ablation study on AFPN

Our third ablation study analyzes the effect of the AFPN feature fusion module. The impact of the AFPN module is shown in the Table 5. The first row of the table presents the segmentation performance of the original HTC. The second row shows the segmentation results using AFPN as the feature extraction module, and the third row displays the experimental results of RACNet based on the aforementioned ablation experiments.Table 5 Results of ablation study on AFPN.

Model	AP50 (%)		Segmentation AP (%)		
Small	Medium	Large	
HTC	44.8	9.3	36.6	42.7	
HTC + AFPN	50.2	11.3	43.1	45.4	
RACNet	50.8	12.3	43.7	45.9	
Significant values are in bold.

Table 5 demonstrates the capturing ability of AFPN for medium-sized targets, with an improvement of 6.5% in APmedium. We visualized the results of the AFPN feature fusion module ablation experiment. The visualization results are shown in Fig. 10.Fig. 10 (a–c) show the test results for three typical scenarios. The first column shows the original RGB image, the second column shows the ground truth labels, and the third and fourth columns show the original HTC network and the visualization results using AFPN as the feature extraction module. The fifth column shows the final results of the complete RACNet.

As can be seen in Fig. 10, the number of missed targets decreases after the model added the AFPN feature fusion module. The experimental results of RACNet are better, although there is still the problem of over-segmentation on small targets.

Conclusion

This paper proposes the RACNet, an improved cascade segmentation network. The RACNet utilizes various strategies for the instance segmentation task of rice in satellite remote sensing images. Firstly, we employ RGB and NDVI as multimodal inputs to the network, supplementing the representation of rice during the flowering period to enhance the category differences with other land features and plants. Secondly, we construct the CAD-Resnet backbone network for feature extraction using the ECA mechanism and DCN convolution. Compared to ResNet-50, our CAD-Resnet backbone network focuses more on contributing channel features, providing an advantage in crop classification tasks with prominent spectral features. At the same time, the dynamic convolution kernel releases a fixed receptive field. It makes the network better able to focus on fragmented fields located in areas of elevation fluctuation. Lastly, in visual tasks for remote sensing images, scale features are crucial. The RACNet network employs the AFPN module to fuse features with scale variance in vegetation, reducing the loss of features between non-adjacent levels and achieving significant improvements in performance metrics.

Author contributions

Conceptualization, Z.L. Y.Z. and L.Z.; methodology, Z.L. and K.Z.; software, B.Y.; validation, Z.L., B.Y.; formal analysis, Z.L.; investigation, Z.L.; resources, Y.Z. and L.Z.; data curation, Z.L. and K.Z.; writing—original draft preparation, Z.L.; writing—review and editing, Y.Z.; visualization, Z.L.; supervision, Y.Z.; project administration, K.Z. and L.Z.; funding acquisition, Y.Z. All authors have read and agreed to the published version of the manuscript.

Funding

No funding was received to assist with the preparation of this manuscript.

Data availability

The data that support the findings of this study are available from Meishan Land and Resources Bureau but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the corresponding author (Yan Zhang, zyan0htzh@gmail.com) upon reasonable request and with permission of Huantian Wisdom Technology Co.,Ltd.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Bin Rahman AR Zhang J Trends in rice research: 2030 and beyond Food Energy Secur. 2023 12 2 e390 10.1002/fes3.390
Bin Rahman, A. R. & Zhang, J. Trends in rice research: 2030 and beyond. Food Energy Secur. 12(2), e390 (2023).10.1002/fes3.390
2. Manjunath K More RS Jain N Panigrahy S Parihar J Mapping of rice-cropping pattern and cultural type using remote-sensing and ancillary data: A case study for South and Southeast Asian countries Int. J. Remote Sens. 2015 36 24 6008 6030 10.1080/01431161.2015.1110259
Manjunath, K., More, R. S., Jain, N., Panigrahy, S. & Parihar, J. Mapping of rice-cropping pattern and cultural type using remote-sensing and ancillary data: A case study for South and Southeast Asian countries. Int. J. Remote Sens. 36(24), 6008–6030 (2015).10.1080/01431161.2015.1110259
3. Hajjar MJ Ahmed N Alhudaib KA Ullah H Integrated insect pest management techniques for rice Sustainability 2023 15 5 4499 10.3390/su15054499
Hajjar, M. J., Ahmed, N., Alhudaib, K. A. & Ullah, H. Integrated insect pest management techniques for rice. Sustainability 15(5), 4499 (2023).10.3390/su15054499
4. Khan SD Basalamah S Lbath A Weed-Crop segmentation in drone images with a novel encoder–decoder framework enhanced via attention modules Remote Sens. 2023 15 23 5615 10.3390/rs15235615
Khan, S. D., Basalamah, S. & Lbath, A. Weed-Crop segmentation in drone images with a novel encoder–decoder framework enhanced via attention modules. Remote Sens. 15(23), 5615 (2023).10.3390/rs15235615
5. Weiss M Jacob F Duveiller G Remote sensing for agricultural applications: A meta-review Remote Sens. Environ. 2020 236 111402 10.1016/j.rse.2019.111402
Weiss, M., Jacob, F. & Duveiller, G. Remote sensing for agricultural applications: A meta-review. Remote Sens. Environ. 236, 111402 (2020).10.1016/j.rse.2019.111402
6. Karmakar P Teng SW Murshed M Pang S Li Y Lin H Crop monitoring by multimodal remote sensing: A review Remote Sens. Appl.: Soc. Environ. 2023 33 101093
Karmakar, P. et al. Crop monitoring by multimodal remote sensing: A review. Remote Sens. Appl.: Soc. Environ. 33, 101093 (2023).
7. Yao J Zhang B Li C Hong D Chanussot J Extended vision transformer (ExViT) for land use and land cover classification: A multimodal deep learning framework IEEE Trans. Geosci. Remote Sens. 2023 61 1 15
Yao, J., Zhang, B., Li, C., Hong, D. & Chanussot, J. Extended vision transformer (ExViT) for land use and land cover classification: A multimodal deep learning framework. IEEE Trans. Geosci. Remote Sens. 61, 1–15 (2023).
8. Wang, J. et al. (2021). Classification of rice yield using UAV-based hyperspectral imagery and lodging feature. Plant Phenomics.
9. Ma J Liu B Ji L Zhu Z Wu Y Jiao W Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery Int. J. Appl. Earth Obs. Geoinf. 2023 118 103292
Ma, J. et al. Field-scale yield prediction of winter wheat under different irrigation regimes based on dynamic fusion of multimodal UAV imagery. Int. J. Appl. Earth Obs. Geoinf. 118, 103292 (2023).
10. Zhou X Zhu X Dong Z Guo W Estimation of biomass in wheat using random forest regression algorithm and remote sensing data Crop J. 2016 4 3 212 219 10.1016/j.cj.2016.01.008
Zhou, X., Zhu, X., Dong, Z. & Guo, W. Estimation of biomass in wheat using random forest regression algorithm and remote sensing data. Crop J. 4(3), 212–219 (2016).10.1016/j.cj.2016.01.008
11. Tariq A Yan J Gagnon AS Riaz Khan M Mumtaz F Mapping of cropland, cropping patterns and crop types by combining optical remote sensing images with decision tree classifier and random forest Geo-Spatial Inf. Sci. 2023 26 3 302 320 10.1080/10095020.2022.2100287
Tariq, A., Yan, J., Gagnon, A. S., Riaz Khan, M. & Mumtaz, F. Mapping of cropland, cropping patterns and crop types by combining optical remote sensing images with decision tree classifier and random forest. Geo-Spatial Inf. Sci. 26(3), 302–320 (2023).10.1080/10095020.2022.2100287
12. Sisodia, P. S., Tiwari, V., & Kumar, A. (2014). Analysis of supervised maximum likelihood classification for remote sensing image. Paper presented at the International conference on recent advances and innovations in engineering (ICRAIE-2014).
13. Yang L Wang L Abubakar GA Huang J High-resolution rice mapping based on SNIC segmentation and multi-source remote sensing images Remote Sens. 2021 13 6 1148 10.3390/rs13061148
Yang, L., Wang, L., Abubakar, G. A. & Huang, J. High-resolution rice mapping based on SNIC segmentation and multi-source remote sensing images. Remote Sens. 13(6), 1148 (2021).10.3390/rs13061148
14. Huang F Comparison of winter wheat extraction methods based on different time series of vegetation indices in the Northeastern margin of the Qinghai-Tibet Plateau: A case study of Minhe China Remote Sens. 2022 14 2 343 10.3390/rs14020343
Huang, F. et al. Comparison of winter wheat extraction methods based on different time series of vegetation indices in the Northeastern margin of the Qinghai-Tibet Plateau: A case study of Minhe China. Remote Sens. 14(2), 343 (2022).10.3390/rs14020343
15. Bolya, D., Zhou, C., Xiao, F., & Lee, Y. J. (2019). Yolact: Real-time instance segmentation. Paper presented at the Proceedings of the IEEE/CVF international conference on computer vision.
16. Mo J Deep learning-based instance segmentation method of litchi canopy from UAV-acquired images Remote Sens. 2021 13 19 3919 10.3390/rs13193919
Mo, J. et al. Deep learning-based instance segmentation method of litchi canopy from UAV-acquired images. Remote Sens. 13(19), 3919 (2021).10.3390/rs13193919
17. He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask r-cnn. Paper presented at the Proceedings of the IEEE international conference on computer vision.
18. Wang, K., Liew, J. H., Zou, Y., Zhou, D., & Feng, J. (2019). Panet: Few-shot image semantic segmentation with prototype alignment. Paper presented at the proceedings of the IEEE/CVF international conference on computer vision.
19. Huang, Z., Huang, L., Gong, Y., Huang, C., & Wang, X. (2019). Mask scoring r-cnn. Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
20. de Albuquerque AO de Carvalho OL eSilva CR Luiz AS de Bem PP Gomes RA Guimarães RF de Carvalho OA Dealing with clouds and seasonal changes for center pivot irrigation systems detection using instance segmentation in sentinel-2 time series IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021 14 8447 8457 10.1109/JSTARS.2021.3104726
de Albuquerque, A. O. et al. Dealing with clouds and seasonal changes for center pivot irrigation systems detection using instance segmentation in sentinel-2 time series. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 14, 8447–8457 (2021).10.1109/JSTARS.2021.3104726
21. Carvalho OL de Carvalho Junior OA Albuquerque AO Bem PP Silva CR Ferreira PH Moura RD Gomes RA Guimaraes RF Borges DL Instance segmentation for large, multi-channel remote sensing imagery using mask-RCNN and a mosaicking approach Remote Sens. 2020 13 1 39 10.3390/rs13010039
Carvalho, O. L. et al. Instance segmentation for large, multi-channel remote sensing imagery using mask-RCNN and a mosaicking approach. Remote Sens. 13(1), 39 (2020).10.3390/rs13010039
22. Chen, K. et al. (2019). Hybrid task cascade for instance segmentation. Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
23. Deng R Zhou M Huang Y Tu W Hybrid task cascade-based building extraction method in remote sensing imagery Remote Sens. 2023 15 20 4907 10.3390/rs15204907
Deng, R., Zhou, M., Huang, Y. & Tu, W. Hybrid task cascade-based building extraction method in remote sensing imagery. Remote Sens. 15(20), 4907 (2023).10.3390/rs15204907
24. Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
25. Wang, Q., Wu, B., Zhu, P., Li, P., Zuo, W., & Hu, Q. (2020). ECA-Net: Efficient channel attention for deep convolutional neural networks. Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
26. Zhou G Liu W Zhu Q Lu Y Liu Y ECA-mobilenetv3 (large)+ SegNet model for binary sugarcane classification of remotely sensed images IEEE Trans. Geosci. Remote Sens. 2022 60 1 15
Zhou, G., Liu, W., Zhu, Q., Lu, Y. & Liu, Y. ECA-mobilenetv3 (large)+ SegNet model for binary sugarcane classification of remotely sensed images. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2022).
27. Khan SD Alarabi L Basalamah S Segmentation of farmlands in aerial images by deep learning framework with feature fusion and context aggregation modules Multimed. Tools Appl. 2023 82 27 42353 42372 10.1007/s11042-023-14962-5
Khan, S. D., Alarabi, L. & Basalamah, S. Segmentation of farmlands in aerial images by deep learning framework with feature fusion and context aggregation modules. Multimed. Tools Appl. 82(27), 42353–42372 (2023).10.1007/s11042-023-14962-5
28. Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
29. Li, H., Zhang, R., Pan, Y., Ren, J., & Shen, F. (2024). Lr-fpn: Enhancing remote sensing object detection with location refined feature pyramid network. arXiv preprint arXiv:2404.01614.
30. Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018). Path aggregation network for instance segmentation. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
31. Yang, G., Lei, J., Zhu, Z., Cheng, S., Feng, Z., & Liang, R. (2023). AFPN: Asymptotic feature pyramid network for object detection. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (pp 2184–2189). IEEE.
