
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)12892-7
10.1016/j.heliyon.2024.e36861
e36861
Research Article
Chinese text classification method based on sentence information enhancement and feature fusion
Zhu Binglin binglinzg@163.com
⁎⁎
Pan Wei panwei@cwnu.edu.cn
⁎
School of Computer Science, China West Normal University, China
⁎ Corresponding author. panwei@cwnu.edu.cn
⁎⁎ Corresponding author. binglinzg@163.com
24 8 2024
15 9 2024
24 8 2024
10 17 e3686128 3 2024
9 8 2024
23 8 2024
© 2024 The Authors
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Text classification involves annotating text data with specific labels and is a crucial research task in the field of natural language processing. Chinese text classification presents significant challenges due to the complex semantics of the language, difficulties in semantic feature extraction, and the interleaving and irregularity of lexical features. Traditional methods often struggle to manage the relationships between words and sentences in Chinese, hindering the model's ability to capture deep semantic information and resulting in poor classification performance. To address these issues, a Chinese text classification method based on utterance information enhancement and feature fusion is proposed. This method first embeds the text into a unified space and obtains feature representations of word vectors and sentence vectors using the BERT (Bidirectional Encoder Representations from Transformers) pre-trained language model. Subsequently, an utterance information enhancement module is constructed to perform syntactic enhancement and feature extraction on the sentence information within the text. Additionally, a feature fusion strategy is introduced to combine the enhanced sentence-level information features with the word-level features extracted by the Bi-GRU (Bidirectional Gated Recurrent Unit network), culminating in the classification output. This approach effectively enhances the feature representation of Chinese text and significantly filters out irrelevant and noisy information. Evaluations on several Chinese datasets demonstrate that the proposed method surpasses existing mainstream classification models in terms of classification accuracy and F1 value, validating its effectiveness and feasibility.

Keywords

Chinese text classification
Sentence-level information enhancement
Feature fusion
Attention mechanism
Deep learning
==== Body
pmc1 Introduction

With the explosive growth of the number of Internet users, the amount of text data has increased dramatically. How to extract valuable information from this text data has become an important research direction in the field of natural language processing. Text classification is a task that assigns predefined categories to texts based on their content. It has a wide range of applications in many fields such as sentiment polarity analysis [1,2], spam recognition [3], news classification [4], information retrieval [5] and rumor detection [6]. In recent years, deep learning has achieved remarkable success in text classification tasks [7]. Deep neural networks have strong feature learning and representation capabilities, and can automatically learn the feature representation of text data to improve the performance of text classification. Examples include CNN (Convolutional Neural Network) model [8,9], RNN (Recurrent Neural Network) model [10,11], and improved Bi-LSTM (Bidirectional Long Short-Term Memory) model based on Long Short-Term Memory [12,13]. Compared with traditional machine learning algorithms such as SVM (Support Vector Machine) [14] and NB (Naive Bayes) [15], the network model based on deep learning can extract text features more fully and has better classification results.

In deep learning, when the text data is consistent, the effect of text classification mainly depends on the feature extraction method and feature representation method. In terms of feature extraction, Kim [16] first applied CNN to text classification and proposed the TextCNN model, which has the advantage of effectively extracting the local features of text. To further improve the classification effect of TextCNN model, Schwenk et al. [17] constructed a deep CNN model by increasing the number of convolutional layers and increasing the convolutional kernel receptive field. However, the experimental results showed that the deep CNN model constructed by increasing the number of layers vertically was not obvious to improve the text classification effect. Later, Er et al. [18] proposed a new pooling scheme, called attention pooling, to retain the most important information in the pooling stage to improve the model's ability in terms of modeling sentence classification. In addition, Johnson et al. [19] proposed a deep convolutional neural network model DPCNN (Deep Pyramid Convolutional Neural Networks) based on the word level, which is advantageous in terms of lower complexity and superior classification results compared to the character level based convolutional neural network model. In addition to Convolutional Neural Networks, Recurrent Neural Network is also often used for text classification. RNN can not only read the current information, but also record the previously read information, which can better capture the long-term dependency of text data [20]. However, traditional RNN models suffer from the problem of gradient vanishing and gradient explosion, which may make the model difficult to train. To improve this problem, Long Short-Term Memory (LSTM) was proposed, and LSTM achieved better results in text classification tasks by changing the structure of the recurrent layer. Liu et al. [21] mitigated some of the problems in RNN models by applying LSTM to text classification models. Although LSTM improves the classification accuracy over RNN, it still has the problem of inadequate extraction of text features, and the model has higher computational complexity and longer training time. In recent years, models based on the Transformer architecture have become the mainstream technology for Chinese text classification. For example, Xu et al. [22] proposed a text classification model that integrates Chinese BERTology-wwm with Graph Convolutional Networks (GCNs), demonstrating superior performance in tasks rich in structured relationships.

In terms of feature representation of text, Mikolov et al. [23] proposed Word2vec, a feature representation model for static word vectors, which can better represent the association information of contextual words and solve the problems of traditional one-hot coding. However, a good linguistic feature representation model should be able to contain rich semantic and syntactic information, and be able to model polysemous words. And the traditional word vector representations are all context-independent, i.e., fixed word vectors. Therefore, Peters et al. [24] proposed the ELMO model, which can obtain dynamic word vectors based on contextual information to solve the problem of multiple meanings of a word. Kant et al. [25] applied the ELMO model to text classification and achieved excellent classification results. Tao et al. [26] developed a model based on cognitive principles that not only focuses on the literal meaning of the text but also attempts to enhance classification accuracy by simulating the reader's cognitive process.

Although deep learning models have achieved better results in text classification tasks. However, for Chinese text classification problems, most existing methods have difficulty in capturing deeper Chinese semantic information. This is because these modeling approaches usually only consider the word-level information of Chinese text, ignoring the sentence-level information that is important for understanding the meaning of Chinese text. In addition, these approaches usually rely on only a single type of features, such as word embeddings or character n-gram features, without explicitly considering the importance of different features. This results in models with weak semantic connections between contexts and difficulty in dealing with problems such as semantic interleaving. Moreover, Chinese text is characterized by complex semantics, sparse and irregular semantic features. If do not consider the linguistic knowledge to link the whole sentence and the contextual relationship, the final classification results obtained by the model are likely to be wrong. To solve the above problems, this paper proposes a new Chinese text classification method based on utterance information enhancement and feature fusion. The main contributions of this research are as follows.(1) Sentence Information Enhancement: To further enhance the sentence information within the text, the study designs multiple feed-forward neural networks combined with residual connections and a multi-head attention mechanism. This approach effectively weights important features within sentences, enhancing the precision of feature extraction and improving the model's ability to capture critical information.

(2) Design of a Parallel Word-Level Feature Extraction Network: The research introduces a parallel word-level feature extraction network that first uses the BERT model to obtain feature representations of words, including information such as word embeddings and POS tags. These representations are then fed into a Bi-GRU network for feature extraction, capturing comprehensive word-level features and improving text classification accuracy.

(3) Feature fusion module: in the feature extraction process, this method weights and fuses the enhanced sentence information features with the extracted word-level features. This feature fusion strategy can effectively integrate different levels of information and improve the classification performance.

In addition, the experimental results on several public datasets and self-made datasets show that the method can effectively extract the deep semantic information of Chinese text and can deal with the dependency relationship between words and sentences. Meanwhile, the method shows good robustness and classification performance in coping with small data samples, large differences in sentence lengths and uneven sample categories. In summary, this research improves Chinese text classification performance by introducing innovative feature enhancement and fusion techniques, providing new insights and methods for research and application in this field.

2 Methods

In this study, we propose a novel Chinese text classification model based on sentence information enhancement and feature fusion. In this model, the original text data is first pre-processed and converted to the corresponding Token representation based on the word list. Then the input text is encoded using BERT and converted into a high-dimensional vector representation to obtain the word embedding and sentence embedding information. Subsequently, the word vectors are learned using Bi-GRU, while the sentence information is enhanced and features are extracted using the utterance information enhancement module. Thus, word-level semantic features and enhanced sentence information features are obtained respectively. Finally, the two features are fused using the weighted average method and fed into the fully connected layer and Softmax to obtain the classification results. The model structure of the method is shown in Fig. 1.Fig. 1 Model structure.

Fig. 1

2.1 Text embedding layer

In the field of natural language processing, the accurate representation of text features plays a decisive role in the quality of the task output. The preprocessing steps for raw text include word slicing, deactivation elimination, and named entity recognition. In this paper, the method adopts word embedding technique to realize the representation of text features, specifically, the input text is divided into independent characters, and then each character is mapped to a unique Token, as show in equation (1):(1) e(wi)∈Rd

Where wi represents the i-th character, e(wi) refers to the corresponding word embedding vector of the character, and Rd denotes the d-dimensional vector space.

In recent years, with the emergence of pre-trained language models like BERT [27], researchers have been able to obtain textual feature representations rich in semantic information. The distinguishing feature of the BERT pre-trained language model is its bi-directional encoder design, which is based on the coding part of the Transformer [28] model. BERT also accepts sentences consisting of more than one word as input. Prior to input, each sentence needs to be preprocessed, a [CLS] marker indicating the beginning of the sentence is added at the beginning of the sentence and a [SEP] marker indicating the end of the sentence is inserted at the end of the sentence. Subsequently, the input text information is converted into three embedding vectors after preprocessing, which are word vector, paragraph vector, and positional embedding vector. The final input vector is the sum of these three embedding vectors, as show in equation (2):(2) V=Echar⨁Eseg⨁Epos

Here V denotes the final input word vector, and Echar, Eseg and Epos represent word vector, segment vector and position vector, respectively. Specifically, in the pre-training process, to determine the sequential relationship between two different sentences, the segment vector embedding is mainly used to pair-wise connect the sentences in the input text through [CLS] markers to distinguish different sentences. Position vector embedding is mainly used to distinguish the differences in semantic information between words at different positions in the text sequence. In the pre-training stage, the BERT model simultaneously considers the information in both the front and back directions in a text sequence, which gives the model the ability to understand long-distance dependencies and to appreciate the universal properties of language. Therefore, by utilizing the BERT model as a text embedding layer, the corresponding word embedding vectors can be obtained to further improve the performance of text classification.

2.2 Sentence information enhancement module

To extract the semantic properties in Chinese utterances more deeply, we build a sentence information enhancement module. The module consists of multiple fully-connected networks integrating residual connections, plus a normalization layer and a multi-head attention mechanism layer. Specifically, the fully-connected layer functions to transform weighted vector representations into higher dimensional vector representations that contain a large amount of information. At the same time, this can also dimensionalize the high-dimensional vector representation down to a low-dimensional vector representation, and this low-dimensional vector representation can capture the key information in the text more effectively. Adopting this approach for Chinese sentence information enhancement can fit the semantic information of the text more accurately. However, too deep fully connected neural networks often lead to model performance degradation and semantic distortion. Therefore, to prevent degradation of the network and to better capture the detailed information in the sentence text, residual connectivity is introduced in this part of the network.

Residual Network (ResNet) [29] is a widely used deep learning technique, and its key idea is to build deep neural networks by constructing residual blocks. Inside the residual network, each residual block contains a Skip Connection, which can pass the input to the output directly, thus alleviating the problems of gradient vanishing and gradient explosion in deep neural networks. The residual network are calculated as shown in equation (3):(3) y=F(x,{wi})+y

Where x denotes the input data, wi denotes the weight of the layer, and F denotes the transform operation function in the residual block. The output y can be obtained by adding x with the transform operation F. This process can be seen as a correction to the input x, which makes it easier for the model to learn the important information in the input x.

There is also a normalization layer and a multi-head attention mechanism in the sentence information enhancement module. The normalization layer plays an extremely important role in the deep learning model. Its main purpose is to normalize the input data into a form with mean 0 and variance 1. This process can lead to faster convergence of the model and can reduce the instability during the training process, thus improving the generalization ability of the model. Another important role of the normalization layer is that it can mitigate the so-called internal covariate shift problem. That is, during the model training process, the input distribution of each layer changes as the weights of the previous layer are updated. This phenomenon can complicate and destabilize the training process. This problem can be mitigated to some extent by introducing a normalization layer, which ensures that the inputs to each layer have the same distribution, thus reducing the impact of outliers on the performance of the network model. When the sentence embedding vector passes through a fully connected neural network with residual connections, the output features are X={x1,x2,x3,…,xn}, where xi is the feature vector with dimension d. The normalization is calculated as shown in equations (4), (5), (6), (7):(4) μ=1d∑xi

(5) σ2=1d∑(xi−μ)2

(6) xˆi=xi−μσ2+ϵ

(7) yi=γxˆi+β

Where μ is the mean σ2 is the variance, equation (5) represents the normalization of each sample feature, where ϵ is a very small positive number, here the value is taken as 1×10−5, which is used to avoid the case where the variance is zero. γ and β are learnable parameters that are automatically updated when the model is backpropagated and are used to scale and flatten the normalized features, which allows the network to be adapted to specific tasks and data to enhance the robustness of the model.

The multi-head attention mechanism is used to weight the output data of the normalized fully-connected layer, which helps the model to better process the input data and capture important information in the text. Also, multi-head attention allows the model to focus on different parts of the input vector at the same time, the model will divide the dimensions of the input vector into multiple heads, each of which computes a set of weights. These weights can then be combined for better extraction of text data features. By introducing the multi-head self-attention mechanism, it not only improves the efficiency of the model, but also increases the expressive power of the model so that the model can better understand and process complex input data. The formula for the multi-head attention mechanism is shown in equations (8), (9), (10):(8) Q=WqX+bq,K=WkX+bk,V=WvX+bv

(9) Attention(Q,K,V)=Softmax(QKTdk)V

(10) MultiHead(Q,K,V)=Concat(head1,head2,…,headh)WO

Where Q, K, and V denote the query vector, key vector, and value vector, respectively, Wq, Wk, and Wv are the corresponding parameter weight matrices, bq, bk, and bv are the corresponding bias terms, and dk denotes the dimension of the key vector. The similarity between Q and K can be obtained by calculating the dot product of Q and K. Then the similarity is converted into weights using the Softmax function, and finally the weights are multiplied with the value vector to obtain the output of this attention mechanism. In the computation of multi-head attention mechanism, h denotes the number of attention heads, headi denotes the output of the i-th head, and WO denotes the learnable weight matrix. The computation process of each attention head is like that of a single attention mechanism computation, but using different query, key, and value matrices. Finally, the output of each head is spliced together and multiplied with the weight matrix to get the final output, which extracts the important features of the enhanced Chinese sentence information.

2.3 Word-level feature extraction network

In addition to the sentence information enhancement module, we also set up a feature extraction network at the word-level. This network achieves nonlinear fitting and feature extraction by feeding the output word vectors of the BERT encoder into the Bi-GRU network. Bi-GRU, or Bidirectional Gated Recurrent Unit, provides a more comprehensive context-awareness capability by considering both past and future information, and can fully extract the semantic information in Chinese words, which in turn improves the expressive ability of the model. Meanwhile, Bi-GRU effectively solves the problem of long-range dependency by introducing the gate mechanism, and its computational efficiency is higher than that of Bi-LSTM, which effectively reduces the complexity of the model, makes the training process more efficient, and at the same time reduces the possibility of overfitting.

In the cell structure of the GRU, xt denotes the input of the current time step, and ht−1 and ht denote the hidden states of the previous time step and the current time step, respectively. In the current time step, the update gate zt will be computed first, which determines how much of the previous moment's hidden state should be considered when computing the current candidate hidden state, as shown in equation (11):(11) zt=σ(Wzxt+Uzht−1+bz)

Then the reset gate rt is computed, which determines how much of the previous moment's hidden state should be forgotten when computing the current candidate hidden state, as shown in equation (12):(12) rt=σ(Wrxt+Urht−1+br)

Next the candidate hidden state h˜t is computed, which is based on the input of the current time step and the hidden state of the previous time step adjusted by the reset gate, as shown in equation (13):(13) h˜t=tanh(Wxt+U(rt⨀ht−1)+b)

Finally, the updated hidden state ht of the current time step is computed, which is a linear interpolation of the hidden state of the previous moment and the current candidate hidden state, and the coefficients of the interpolation are determined by the update gate zt with shown in equation (14):(14) ht=(1−zt)⨀ht−1+zt⨀h˜t

In the above equation, W and U denote the learnable parameters of the corresponding step, b is the bias term of the corresponding step, σ is the sigmoid activation function, and ⨀ denotes the element-by-element multiplication. Two independent GRUs are included in the network structure of Bi-GRU, one is responsible for processing forward information and the other is responsible for processing backward information, without any other changes in the internal structure. The two GRUs of Bi-GRU use the same update process and the internal structure remains the same, but the direction of the input sequence is different, the update process is shown in equation (15), (16), (17):(15) ht→=f→(ht−1→,xt)

(16) ht←=f←(ht−1←,xt)

(17) ht=[ht→,ht←]

Where f→ and f← represent the update functions of the forward and backward GRUs, respectively, ht→ and ht← are the hidden states of the forward and backward GRUs at time step t, respectively, and ht is the final output of the Bi-GRU at time step t. In a single GRU, the network can only process the information step by step from the past to the future, and such processing makes the model ineffective in handling some tasks that need to utilize future information. Therefore, in this paper, this problem is solved by introducing a parallel Bi-GRU network, which enables the word-level feature extraction network to obtain more comprehensive information about word-sense features.

2.4 Feature fusion and output layer

Through the feature extraction of the word-level feature extraction network and the sentence information enhancement module, the word-level feature vector Fw and the sentence-level feature vector Fs of the text are obtained respectively. Then, these two feature vectors are weighted and fused to obtain the combined feature vector F. Finally, the feature vector F is input into the fully connected layer and Softmax to obtain the predicted probability P of the text data on each category, and the category with the highest predicted probability value is the final predicted category. Here, P is calculated as shown in equation (18):(18) P=Softmax[Wf(WwFw+WsFs)+bf]

Here, Wf and bf are the weight matrix and bias vector of the fully connected layer, and Ww and Ws are the weighting coefficients of the word-level feature vector Fw and the sentence-level feature vector Fs, respectively.

3 Experiments and analysis of results

In this section, we describe the relevant settings for the experiments, as well as conduct comprehensive experiments to evaluate the effectiveness and applicability of the proposed Chinese text categorization method. We also perform ablation studies to understand the contribution of each component in the model.

3.1 Datasets and evaluation indicators

To verify the effectiveness and applicability of the proposed model in this paper, experiments are conducted on three publicly available Chinese datasets. These three publicly available Chinese datasets are the hotel review dataset (CHR), the take-out review dataset (Waimai) and the medical intent recognition dataset (cMedIC). A Chinese online shopping review dataset (COSR) was also created for experiments to evaluate the generalization of the text-based methods. CHR is a Chinese categorization dataset with data from users' hotel reviews on a platform, with a balanced distribution of positive and negative review samples. Waimai is a Chinese categorical dataset with data from user reviews collected by a takeaway platform. It is a binary labeled dataset with 4000 positive and 7980 negative items, and the number of negative samples is significantly larger than that of positive samples. This may lead to poor performance of the model biased towards positive sentiment classification, a Chinese dataset with severe category imbalance. cMedIC is a Chinese medical text categorization dataset. The dataset consists of queries with three intent labels (no intent, weak intent, and firm intent) and is a small sample dataset. COSR is a self-made Chinese online shopping review dataset to validate the generalization of this paper's method, which contains reviews of ten kinds of commodities, such as electrical appliances, clothing, furniture, and mobile devices, with the ratio of positive and negative categories about 6:4. And the four Chinese datasets selected in this paper contain both short text and medium-length text, and the imbalance of category ratio and staggered sentence length in the datasets also exists, so that the adaptability of this paper's method to categorize different Chinese texts can be fully verified. Fig. 2 show the analysis of the number of sentence lengths for the training set in the four datasets.Fig. 2 Sentence length count analysis of the training set.

Fig. 2

Since there is no separate validation set in the original dataset, this experiment adopts the following strategy for experimental evaluation. First, the original training set was divided into a new training set and validation set in the ratio of 8:2. Specifically, 80 % of the data is used for model training, and the remaining 20 % of the data is used as the validation set to evaluate the performance of the model during the training process. During each round of training, we use the validation set to validate the model. It is worth noting that the model parameters are not updated during the validation phase. The purpose of this process is to monitor the performance of the model on the validation set and thus avoid overfitting. In addition, we adopt an early stopping strategy, i.e., we terminate the training early when the performance of the validation set is no longer improving and save the final weights of the current model. Specifically, we terminate training when the performance metrics of the validation set (e.g., validation loss) no longer improve over several consecutive training rounds to preserve the best model parameters. The final dataset division is shown in Table 1.Table 1 Experimental datasets.

Table 1Dataset	Language	Labels	Train	Dev	Test	
CHR	Chinese	2	7680	1920	1200	
Waimai	Chinese	2	8800	2200	980	
cMedIc	Chinese	3	1347	336	123	
COSR	Chinese	2	10127	2531	1600	

In terms of evaluation indexes, the accuracy (Accuracy) and F1 value (F1-score) are selected to assess the performance of the classification model in this paper. According to the confounding matrix, the experimental results can be categorized into four cases: true positive (TP), false positive (FP), true negative (TN) and false negative (FN). Among them, TP denotes the number of texts whose actual label is positive and the prediction result is also positive; FP denotes the number of texts whose actual label is negative but the prediction result is positive; FN denotes the number of texts whose actual label is positive but the prediction result is negative; and TN denotes the number of texts whose actual label is negative and the prediction result is also negative. The formulas for accuracy and F1 value are as shown in equation (19), (20):(19) Accuracy=TP+TNTP+FP+TN+FN

(20) F1=2*TP2*TP+FP+FN

From the above formula, the accuracy rate is the ratio of the number of samples correctly classified by the classifier to the total number of samples, which intuitively reflects the overall performance of the classifier. However, when there is a category imbalance in the dataset, the accuracy rate may be distorted because the classifier may tend to predict the category with a higher number of samples. Therefore, the F1 value is also chosen as an evaluation metric, which is the reconciled average of precision and recall, and can combine the prediction accuracy and recognition ability of the classifier.

3.2 Parameter settings

The main experimental parameters of the model of this study were set as shown in Table 2.Table 2 Model parameter settings.

Table 2Parameter	Value	
Max_length	256	
Batch_size	64	
Resnet_mlp_size	(128, 64, 128)	
GRU_hidden_size	64	
Learning_rate	5 × 10−4	
Dropout	0.5	
Epochs	30	
Optimizer	Adam	
Activation Function	ReLU	

3.3 Results

The experimental results of the proposed method models as well as the comparison models on four Chinese test datasets are shown in Table 3. For the comparison models, both Naive Bayes (NB) and Support Vector Machine (SVM) use TF-IDF vectors as the text feature extraction method. The parameters for generating the TF-IDF vectors include a minimum document frequency of 1 and a maximum document frequency of 1.0, with no maximum features limit, ensuring that all terms are considered. The kernel function for SVM is set to the Gaussian (RBF) kernel. The Multi-Layer Perceptrons (MLPs) are configured as traditional multilayer perceptron models, containing 2 hidden layers with dimensions of 256 and 128, respectively, and a maximum of 50 training epochs. The word embedding dimension for both TextCNN and Bi-LSTM models is uniformly set to 300, with the word list constructed using Jieba participle. TextCNN employs multiple convolutional kernel sizes of 3, 4, and 5, with 64 convolutional kernels for each size. The Bi-LSTM model comprises 2 hidden layers, each with a dimension of 128.Table 3 Comparison of experimental results.

Table 3Model	CHR	Waimai	cMedIC	COSR	
Acc	F1	Acc	F1	Acc	F1	Acc	F1	
NB	0.759	0.753	0.802	0.793	0.634	0.492	0.702	0.700	
SVM	0.769	0.763	0.783	0.771	0.632	0.493	0.651	0.644	
MLPs	0.753	0.749	0.788	0.791	0.654	0.522	0.665	0.658	
TextCNN	0.822	0.820	0.871	0.846	0.829	0.757	0.815	0.814	
Bi-LSTM	0.861	0.860	0.881	0.863	0.834	0.758	0.833	0.833	
BERT	0.885	0.888	0.902	0.900	0.887	0.873	0.878	0.881	
Ours	0.926	0.926	0.913	0.912	0.918	0.916	0.888	0.898	

From the experimental results, the traditional machine learning methods based on statistics outperform MPLs on some of the datasets. This is because the statistically based methods also achieve better classification performance when the dataset features are easily distinguishable and the semantic features are simpler. Moreover, traditional machine learning methods also have the advantage of being computationally fast, and it does not need to spend a lot of time to train the model as deep learning methods do. However, it also has some fatal flaws, such as generalization performance is average, when the samples have too many label categories and data imbalance, it cannot better extract the semantic features of the text. And the reason why the classification effect of MPLs is weaker than traditional machine learning on some datasets may be because the number of hidden layers is too shallow or the number of neurons is too small, etc., which will lead to underfitting of the model, and the semantic features of the text cannot be adequately learned on datasets with complex sample data. In addition, on the cMedIc dataset the difference between the accuracy and the F1 value of some of the method models is too large, this is because this dataset is a multiclass intent recognition problem with a small number of samples and short text content, which makes it difficult for some of the models to extract the semantic features of the text. Whereas, the experimental results of the method proposed in this paper outperform the comparison models on all four datasets, where the accuracy is 4.1 %, 1.1 %, 3.1 %, and 1 % higher than the best BERT among the benchmark models, and the F1 values are 3.8 %, 1.2 %, 4.3 %, and 1.7 % higher, respectively. The enhancement effect of the textual method is especially obvious on both CHR and cMedIc datasets, which indicates that the sentence information enhancement module and the feature fusion module play a key role in semantic feature extraction for small samples of short texts. When the semantic information is complex and the features are sparse and interleaved this paper's method is also effective in mining the deep semantic information of the text and has excellent classification performance and generalization.

3.4 Ablation experiments

To explore the effect of different modules on the overall performance of the model, this paper conducts ablation experiments, and the experimental results are shown in Table 4. Where Tch is the word-level feature extraction module, SenEn(Res) is the sentence information enhancement module after removing the multiple attention mechanism, SenEn(Attn) is the sentence information enhancement module after removing the residual network, and SenEn(RA) is the complete sentence information enhancement module.Table 4 Results of ablation experiment.

Table 4Model	CHR	Waimai	cMedIC	COSR	
Acc	F1	Acc	F1	Acc	F1	Acc	F1	
+Tch	0.911	0.911	0.903	0.902	0.894	0.887	0.880	0.881	
+SenEn(Res)	0.896	0.896	0.901	0.900	0.894	0.872	0.876	0.875	
+SenEn(Attn)	0.905	0.904	0.902	0.902	0.902	0.896	0.878	0.876	
+SenEn(RA)	0.908	0.908	0.903	0.903	0.910	0.907	0.879	0.877	
+Tch + SenEn(RA)	0.926	0.926	0.913	0.912	0.918	0.916	0.888	0.898	

The ablation experiments conducted to evaluate the proposed model reveal the significant contributions of its components. The combination of Tch, which focuses on word-level feature extraction using Bi-GRU, and SenEn(RA), an enhanced module for complete sentence information, consistently yields the highest performance across all datasets. The Tch module's ability to capture detailed word semantics is crucial, as evidenced by consistent improvements in accuracy and F1 scores when it is included. The exclusion of the attention mechanism (SenEn(Res)) or the residual network (SenEn(Attn)) from the sentence enhancement module results in a noticeable decline in performance, underscoring the importance of these elements for effective feature weighting and gradient flow, respectively. The complete SenEn(RA) module, which integrates both residual connections and the attention mechanism, significantly enhances sentence-level information capture, thereby boosting overall model performance. The integration of Tch and SenEn(RA) modules leverages the strengths of both word-level and sentence-level feature extraction, resulting in comprehensive semantic representation. This fusion strategy shows robustness and effectiveness on all the four datasets. The accuracy of 0.926 and the score of 0.926 F1 are the highest on the CHR dataset, and the accuracy and F1 values of the unbalanced problem on the Waimai dataset are 0.913 and 0.912, respectively. In the cMedIC dataset, the accuracy and F1 values were 0.918 and 0.916, and in the COSR dataset, the accuracy and F1 values were 0.888 and 0.898, respectively, to identify subtle medical intent. These results demonstrate the universality and robustness of the model. It is worth noting that the improvement of the proposed method on the self-made dataset COSR is not obvious, because the COSR dataset is characterized by diversified online shopping comments, involving a wide range of expression modes and informal language. This demonstrates the ongoing challenge of methodological models in the context of informal and changing language patterns, and the need to enhance the model's adaptive mechanisms. Although each component is effective on its own, their integration is essential to maximize performance. The robustness of the model can be attributed to the synergistic effect of combining detailed word-level semantics with high-level sentence-level enhancements. Future research will focus on further refining these components to enhance their adaptability and efficiency. By exploring more complex attention mechanisms and optimizing residual connections, the model can become more robust and capable of handling a wider range of text classification challenges.

4 Conclusion

This study proposes a novel Chinese text classification method that leverages sentence information enhancement and feature fusion to improve feature extraction compared to traditional deep neural network-based approaches. By effectively utilizing both syntactic and lexical information, our method enhances text representation through the integration of semantic and syntactic sentence information, followed by the fusion of lexical features for classification output. This approach captures deep semantic information, thereby enhancing model performance. Experimental results on both benchmark and custom datasets demonstrate the method's superiority in classification performance, stability, and generalization compared to existing mainstream methods. The proposed method advances the field of Chinese text classification by addressing unique challenges such as complex semantics and interleaved lexical features. However, while the method was tested on several datasets, further validation on additional datasets from different domains with varying characteristics is necessary to confirm its generalizability. The increased complexity of integrating multiple modules (BERT, Bi-GRU, and enhancement modules) also leads to higher computational costs. Future research will focus on optimizing the model for faster training and inference, exploring more effective methods to enhance sentence-level information, and integrating diverse types of features. Addressing these limitations by expanding the range of datasets and conducting more extensive comparative analyses will be prioritized. Enhancing the model's computational efficiency will also be a key area of focus to make it more viable for large-scale applications.

Funding

This work was supported by the Natural Science Foundation of China West Normal University (17E064 ). The authors thank the anonymous reviewers for their valuable comments and suggestions.

Data availability statement

Datasets supporting the results of this study are private, and raw experimental data sets may be obtained from the corresponding author upon reasonable request if data for this study is required.

CRediT authorship contribution statement

Binglin Zhu: Writing – original draft, Visualization, Validation, Project administration, Methodology, Data curation. Wei Pan: Supervision, Project administration, Investigation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
==== Refs
References

1 Lu Q. Zhu Z. Xu F. Zhang D. Wu W. Guo Q. Bi-GRU sentiment classification for Chinese based on grammar rules and BERT Int. J. Comput. Intell. Syst. 13 2020 538 548
2 Zhao M. Yang J. Zhang J. Wang S.P. Aggregated graph convolutional networks for aspect-based sentiment classification Inf. Sci. 600 2022 73 93
3 Rao S. Verma A.K. Bhatia T. A review on social spam detection: challenges, open issues, and future directions Expert Syst. Appl. 186 2021 115742
4 Choudhary M. Chouhan S.S. Pilli E.S. Vipparthi S.K. BerConvoNet: a deep learning framework for fake news classification Appl. Soft Comput. 110 2021 107614
5 Mitra B. Craswell N. An introduction to neural information retrieval Found Trends Inf Retr 13 2018 1 126
6 Asghar M.Z. Habib A. Habib A. Khan A. Ali R. Khattak A.M. Exploring deep neural networks for rumor detection J. Ambient Intell. Hum. Comput. 12 2019 4315 4333
7 Minaee S. Cambria E. Gao J. Deep Learning Based Text Classification: A Comprehensive Review 2021
8 Zhou Y. Li J. Chi J. Tang W. Zheng Y. Set-CNN: a text convolutional neural network based on semantic extension for short text classification Knowl. Based Syst 257 2022 109948
9 Soni S. Chouhan S.S. Rathore S.S. TextConvoNet: a convolutional neural network based architecture for text classification Appl. Intell. 53 2022 14249 14268
10 Du J. Vong C.-M. Chen C.L.P. Novel efficient RNN and LSTM-like architectures: recurrent and gated broad learning systems and their applications for text classification IEEE Trans. Cybern. 51 2020 1586 1597
11 Banerjee I. Ling Y. Chen M.C. Hasan S.A. Langlotz C. Moradzadeh N. Lungren M.P. Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification Artif. Intell. Med. 97 2019 79 88 30477892
12 Li W. Qi F. Tang M. Yu Z. Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification Neurocomputing 387 2020 63 77
13 Liu Y. He M. Shi M. Jeon S. A novel model combining transformer and Bi-LSTM for news categorization IEEE Transactions on Computational Social Systems 11 4 2022 4862 4869
14 Dong S. Multi class SVM algorithm with active learning for network traffic classification Expert Syst. Appl. 176 2021 114885
15 Valdiviezo-Diaz P. Ortega F. Cobos E. Lara-Cabrera R. A collaborative filtering approach based on naïve Bayes classifier IEEE Access 7 2019 108581 108592
16 Kim Y. Convolutional neural networks for sentence classification Conference on Empirical Methods in Natural Language Processing 2014
17 Schwenk H. Barrault L. Conneau A. LeCun Y. Very deep convolutional networks for text classification Conference of the European Chapter of the Association for Computational Linguistics 2016
18 Er M.J. Zhang Y. Wang N. Pratama M. Attention pooling-based convolutional neural network for sentence modelling Inf. Sci. 373 2016 388 403
19 Johnson R. Zhang T. Deep Pyramid Convolutional Neural Networks for Text Categorization 2017 Annual Meeting of the Association for Computational Linguistics
20 Hu H. Liao M. Zhang C. Jing Y. Text classification based recurrent neural network 2020 IEEE 5th Information Technology and Mechatronics Engineering Conference (ITOEC) 2020 652 655
21 Liu G. Guo J. Bidirectional LSTM with attention mechanism and convolutional layer for text classification Neurocomputing 337 2019 325 338
22 Xu X. Chang Y. An J. Chinese text classification by combining Chinese-BERTology-wwm and GCN PeerJ Computer Science 9 2023 e1544
23 Mikolov T. Sutskever I. Chen K. Corrado G.S. Dean J. Distributed representations of words and phrases and their compositionality Neural Information Processing Systems 26 2013
24 Peters M.E. Neumann M. Iyyer M. Gardner M. Clark C. Lee K. Zettlemoyer L. Deep Contextualized Word Representations 2018 ArXiv abs/1802.05365
25 Kant N. Puri R. Yakovenko N. Catanzaro B. Practical Text Classification with Large Pre-trained Language Models 2018 ArXiv abs/1812.01207
26 Tao H. Zhu G. Chen E. Learning from ideography and labels: a schema-aware radical-guided associative model for Chinese text classification IEEE Trans. Knowl. Data Eng. 35 6 2022 6043 6057
27 Devlin J. Chang M.-W. Lee K. Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding 2019 North American Chapter of the Association for Computational Linguistics
28 Vaswani A. Shazeer N.M. Parmar N. Uszkoreit J. Jones L. Gomez A.N. Polosukhin I. Attention is all you need Neural Information Processing Systems 2017 5998 6008
29 He K. Zhang X. Ren S. Sun J. Deep residual learning for image recognition 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015 770 778
