
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)13198-2
10.1016/j.heliyon.2024.e37167
e37167
Research Article
Graphic association learning: Multimodal feature extraction and fusion of image and text using artificial intelligence techniques
Lu Guangyun lugy5@163.com
a
Ni Zhiping nizhiping2023@163.com
a⁎
Wei Ling wl113276310@outlook.com
a⁎
Cheng Junwei 15507107584@163.com
a
Huang Wei huang_w1@sina.com
b
a College of Information Science and Engineering, Liuzhou Institute of Technology, 545616, Liuzhou, Guangxi, China
b College of automotive Engineering, Liuzhou Institute of Technology, 545616, Liuzhou, Guangxi, China
⁎ Corresponding authors. nizhiping2023@163.comwl113276310@outlook.com
30 8 2024
30 9 2024
30 8 2024
10 18 e3716723 4 2024
16 8 2024
28 8 2024
© 2024 The Authors
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
With the advancement of technology in recent years, the application of artificial intelligence in real life has become more extensive. Graphic recognition is a hot spot in the current research of related technologies. It involves machines extracting key information from pictures and combining it with natural language processing for in-depth understanding. Existing methods still have obvious deficiencies in fine-grained recognition and deep understanding of contextual context. Addressing these issues to achieve high-quality image-text recognition is crucial for various application scenarios, such as accessibility technologies, content creation, and virtual assistants. To tackle this challenge, a novel approach is proposed that combines the Mask R-CNN, DCGAN, and ALBERT models. Specifically, the Mask R-CNN specializes in high-precision image recognition and segmentation, the DCGAN captures and generates nuanced features from images, and the ALBERT model is responsible for deep natural language processing and semantic understanding of this visual information. Experimental results clearly validate the superiority of this method. Compared to traditional image-text recognition techniques, the recognition accuracy is improved from 85.3% to 92.5%, and performance in contextual and situational understanding is enhanced. The advancement of this technology has far-reaching implications for research in machine vision and natural language processing and open new possibilities for practical applications.

Keywords

Text matching
Image matching
ALBERT
Mask R-CNN
DCGAN
Multimodal feature
Graphic association
==== Body
pmc1 Introduction

In recent years, there has been a clear popular trend in the field of computer vision and natural language processing, with researchers actively exploring ways to integrate visual recognition and natural language processing techniques to improve performance on multimodal tasks [1]. The central goal of this trend is to create systems that make it easier to understand and generate information containing both visual content and natural language descriptions [2]. Current research focuses on various areas, such as graphic recognition, multimodal learning, sentiment analysis, cross-media retrieval, applications of Generative Adversarial Networks (GAN), automatic image annotation, and multilingual multimodal research [3]. These research directions represent current hotspots that have attracted widespread attention and are expected to promote AI development, enrich multimedia content processing, and improve user interaction experiences [4].

Several studies have made significant contributions to the integration of visual recognition and natural language processing. For instance, VisualBERT and VilBERT primarily enhance text processing with visual information but lack robust image generation and precise object segmentation capabilities [5][6][7]. On the other hand, models like Show and Tell and Show, Attend and Tell have made strides in generating textual descriptions from images but do not fully exploit the bidirectional nature of multimodal tasks. More recently, works such as Oscar and ClipCap have advanced image captioning but still face challenges in handling large-scale unlabeled data and enhancing model interpretability and robustness [8][9].

Despite these advancements, several challenges remain. Accurate alignment of visual and verbal information is crucial for ensuring that multimodal data work together effectively [10]. Handling large amounts of unlabeled data remains a significant hurdle, requiring effective automatic labeling and learning methods. Furthermore, model interpretability and robustness are urgent issues that need to be addressed to ensure the credibility and robustness of these fusion models in real-world applications [11][12]. Overcoming these challenges will provide a broader perspective for future multimodal research development.

To address these challenges and bridge existing gaps in visual recognition and natural language processing techniques, we propose the ALBERT-DCGAN-Mask R-CNN network. This innovative model for graphic recognition consists of three main components, each with a unique role. ALBERT is responsible for processing and understanding textual data, utilizing its efficient parameter sharing and sentence order prediction capabilities to analyze semantic relationships deeply. DCGAN processes image data through deep convolutional networks to learn the distributional properties of images and extract key visual features effectively. Mask R-CNN focuses on object detection and segmentation in images, accurately recognizing and analyzing each object in an image. The fusion of these components improves the model's efficiency in processing graphical data and greatly enhances recognition accuracy.

Our model specifically addresses the shortcomings of existing visual recognition and natural language processing techniques in the following ways: enhanced alignment of multimodal data, effective handling of unlabeled data, and improved interpretability and robustness. The core strength of this integrated model lies in its ability to effectively integrate and analyze text and image information. ALBERT provides a strong foundation for language understanding, while DCGAN and Mask R-CNN ensure that image content is accurately parsed. This synergistic approach makes this model more efficient and accurate in dealing with complex graphic-text relationships. Additionally, this fusion model offers wide-ranging application potentials, such as automatic graphic editing, multimedia information retrieval, and AI-assisted teaching.

The core strength of this integrated model lies in its ability to effectively integrate and analyze text and image information. ALBERT provides a strong foundation for language understanding, while DCGAN and Mask R-CNN ensure that image content is accurately parsed. This synergistic approach makes this model more efficient and accurate in dealing with complex graphic-text relationships. Additionally, this fusion model offers wide-ranging application potentials, such as automatic graphic editing, multimedia information retrieval, and AI-assisted teaching. Overall, the ALBERT-DCGAN-Mask R-CNN network not only achieves effective fusion of visual recognition and natural language processing but also shows strong potential and significant advantages in practical applications.

The contribution points of this paper are as follows:• Innovative Design of a Multimodal Fusion Model: We proposed and designed an innovative multimodal fusion model that combines ALBERT, Mask R-CNN, and DCGAN to achieve more accurate and higher-quality image-text recognition. The construction of this model is not only methodologically innovative but also integrates best practices from different domains, providing a new paradigm for multimodal tasks.

• Improvement in the Accuracy and Quality of Image-Text Recognition: By using ALBERT for natural language processing, Mask R-CNN for visual recognition, and DCGAN for image generation, the experiment successfully enhanced the accuracy and quality of image-text recognition tasks. This has practical applications in multimedia content processing and offers significant performance improvements in the field of multimodal tasks.

• By developing innovative models, our research provides more accurate and high-quality solutions for graphic recognition tasks, which are of great practical importance for a wide range of application scenarios, such as multimedia content processing and automated image recognition. These contributions are expected to advance the real-world applications of multimodal technologies and increase the practical value of AI systems.

2 Related work

2.1 Visual attention with recurrent neural networks

Visual Attention with Recurrent Neural Networks (RNNs) is a multimodal model widely used in the field of graphic recognition and image description generation [13]. The core idea of this model is to combine visual attention mechanisms with RNNs so that the model can efficiently process associations between images and text. Previous research has looked at employing visual attention mechanisms with RNNs to deal with the association of images and text [14]. Among them, Qin et al. [15] proposed a model that combines CNN and LSTM to achieve dynamic attention to image regions.

The visual attention mechanism is one of the key components of the model. The visual attention mechanism enables the model to focus on or emphasize specific regions of the image while processing the image. It mimics the attentional processes in the human visual system and enables the model to focus on the most important parts of the image. This mechanism is usually achieved by introducing a distribution of weights in the image that indicate the relative importance of each image region [16]. Another key component of the model is a recurrent neural network, typically a Long Short-Term Memory Network (LSTM) or a Gated Recurrent Unit (GRU). RNNs are responsible for processing text sequences in a graphic recognition task, where they process textual information in a recursive manner, understanding the semantics in the text through context.

This approach achieved impressive results on the task of image description generation. By introducing the attention mechanism, the model is able to focus on the important parts of the image, but it should be noted that the visual attention mechanism requires the computation of the visual attention weights at each time step, which may lead to an increase in the computational complexity of the model, especially when dealing with large-scale datasets. In addition although RNNs have the ability to process sequential data, they are prone to gradient vanishing or gradient explosion problems when processing long sequences, and therefore may not be suitable for tasks that require longer context dependencies.

2.2 Integration of convolutional and recurrent neural networks

Another research work presented by Zhang et al. [17] explores a hybrid approach of Convolutional Neural Networks (CNN) and RNN for image description generation task. CNN and RNN are two commonly used neural network architectures, each of which has achieved significant results in different fields [18]. Convolutional Neural Networks are primarily used to process grid-like data, such as images, while Recurrent Neural Networks are specifically designed to process sequential data, such as natural language [19]. When dealing with multimodal data such as fused images and text, combining CNNs and RNNs can better capture the associations between the data, which is known as Convolutional Neural Networks and Recurrent Neural Networks Fusion (CNN-RNN Fusion).

The fusion model can combine image feature extraction with sequence generation to achieve impressive image annotation results. However, this model still suffers from some information loss when dealing with multimodal data, especially for complex scenes and concepts [20]. Moreover, models fusing CNNs and RNNs are usually more complex than single-modal models, thus requiring more computational resources and time for training and inference, increasing the complexity of model design and debugging.

2.3 Generative adversarial networks

Generative Adversarial Networks (GANs) are a powerful and flexible deep learning model [21]. GANs are unique because they consist of two neural networks: a Generator and a Discriminator, which compete against each other to generate realistic data.

The Generator is usually built as an inverse convolutional neural network (also known as a transposed convolutional network), which learns how to map input noise to realistic data through backpropagation. The generator's task is to generate realistic data such as images, audio, etc [22]. It receives a random noise vector (usually a Gaussian distributed random number) as input and tries to transform this noise vector into data that resembles the real data [18]. The discriminator is usually a convolutional neural network that learns the features of the data and makes classification decisions to evaluate the probability that the given data is real [23]. The discriminator receives either data generated by the generator or real data as input and outputs a probability between 0 and 1 indicating the probability that the input data is real data.

Generative Adversarial Networks have the ability to generate high quality and realistic data such as images, audio and text, which is useful for tasks such as image generation, synthesized audio, text generation, etc., and provides a new approach to data generation. In addition, GANs are trained unsupervised without labelling the data [24]; the generator learns how to generate data from noise, while the discriminator learns how to distinguish between real data and generated data. This gives GANs a distinct advantage over other deep learning models, as obtaining labelled data is often difficult and expensive.

However, the training of GANs is usually more unstable and can suffer from problems such as mode collapse, which causes the generator to learn only part of the pattern of the data. In addition, there is no clear criterion to measure the quality of the generated data, which usually needs to be judged by manual evaluation or other metrics [25]. Moreover, GANs require a lot of computational resources and time to operate, which is especially significant when dealing with high-resolution images or complex data. However, as technology continues to advance, these drawbacks are expected to be gradually overcome, making GANS more powerful.

2.4 Graph convolutional networks

Graph Convolutional Neural Networks (GCNs) are a deep learning model specifically designed to process graph data, such as social networks, recommender systems, and protein interaction networks in bioinformatics [26]. Its emergence has greatly expanded the scope of applications in the field of deep learning, enabling it to efficiently process unstructured data, which is no longer limited to traditional image and text data.

In recent studies, some scholars have started to introduce GCNs to graph recognition tasks [27]. These models aim to use graph structures to model the relationship between images and text. The core idea of GCNs is to propagate information through connections between nodes, an idea inspired by CNNs, but GCNs are specifically designed to work with graph data [28]. In a GCN, each node aggregates information from its neighboring nodes to update its own representation. This process is achieved with the help of a weight matrix, similar to the role of a convolutional kernel in a convolutional neural network [29]. The goal of a GCN is to learn a low-dimensional representation of each node in order to capture the relationships and features between nodes [30].

However, despite their great potential for processing data, GCNs still face some challenges. One of them is how to efficiently construct graph structures so that GCNs can capture key information in the data. Another challenge is the computational complexity of GCNs [31], especially on large-scale datasets, which requires further performance optimization. In addition, GCNs may not be flexible enough for image feature extraction in some scenarios, so more improved and extended variants are needed to cope with different types of graph data and more challenging tasks.

3 Methodology

Our proposed ALBERT-DCGAN-Mask R-CNN model stands out in the field of multimodal fusion for several reasons. Firstly, the integration of ALBERT, DCGAN, and Mask R-CNN into a single framework is novel, leveraging the strengths of each component to enhance overall performance. ALBERT's lightweight yet powerful natural language processing capabilities, DCGAN's advanced image generation, and Mask R-CNN's precise object detection and segmentation collectively address the challenges of multimodal data alignment, large-scale unlabeled data processing, and model interpretability.

Compared to existing work, our approach offers significant improvements. Traditional models often focus on either visual or textual data, failing to achieve seamless integration. For instance, models like ViLT and SimVLM primarily enhance text processing with visual information but lack robust image generation and precise object segmentation capabilities [5][6]. In contrast, our model not only processes and aligns multimodal data but also generates high-quality images that contribute to better recognition and understanding.

Furthermore, while previous efforts such as image-captioning models (e.g., Oscar and ClipCap) have made strides in generating textual descriptions from images, they do not fully exploit the bidirectional nature of multimodal tasks [8][9]. Our model addresses this gap by ensuring robust image-to-text and text-to-image matching, thereby enhancing the overall capability of multimodal information processing.

In summary, the novelty of our proposed model lies in its comprehensive approach to multimodal fusion, combining cutting-edge techniques from both computer vision and natural language processing to achieve superior performance in image-text recognition and matching tasks.

3.1 Overview of our network

The overall structure of this model is shown in the Fig. 1, which covers the whole process from pre-training to subsequent training to similarity metrics. The ALBERT-DCGAN-Mask R-CNN model skillfully combines image recognition and natural language processing techniques through a series of well-designed steps to accomplish the task of image recognition. In the pre-training phase, a basic recognition framework is constructed using labeled training images and real samples with the help of DCGAN and the natural language processing model ALBERT. Subsequently, the model is trained to improve the recognition ability. The test samples are passed through Mask R-CNN for object detection and classification, and the results are further optimized through adaptive polling. The system comprehensively analyzes text-to-image and image-to-text similarities through semantic filtering and global similarity metrics to provide the necessary foundation for the model to understand the text and achieve efficient multimodal information processing capabilities.Figure 1 Image recognition system process framework.

Figure 1

On the left side of the structure diagram is the pre-trained Mask R-CNN, tasked with detecting salient regions in the input image and extracting features from these image regions. The Mask R-CNN is tested using these generated images and real samples to recognize and classify objects in the image, providing a structured understanding of the image. This process is the recognition phase, where the model learns to recognize and classify patterns and features in the image, laying the groundwork for subsequent text matching. Adaptive pooling techniques are then applied to convert the results of recognition and classification into a uniform format, allowing the image data to be effectively compared with the text data.

Subsequently, we introduce the extracted image region features and text features into two different branches. The objective of one branch is to compute the local similarity, and the other is to compute the global similarity of the image-text match. In the local similarity branch, we introduce a powerful semantic filtering module, which matches image regions and words with each other through a cross-attention mechanism and a semantic filtering mechanism, ensuring that the image content is semantically consistent with the text content. This module utilizes ALBERT's text comprehension capabilities in conjunction with Mask R-CNN's image recognition results to filter out image content that does not match the text description, determining the local similarity between image regions and text, including text-to-image and image-to-text similarity. Conversely, our global similarity branch focuses on modeling the association relationships between image regions using a self-attention mechanism to model global associations. Additionally, we employ a learnable pooling module to efficiently aggregate local features and obtain global features of images and texts, and then compute the global similarity between the two.

Among them, the structure of the adaptive pooling module and the semantic filtering module is shown in Fig. 2.Figure 2 The Adaptive Pooling and Semantic Filtering module.

Figure 2

Finally, the model evaluates the degree of image-text matching through multilevel similarity measures, including text-to-image, image-to-text, and global text similarity. The combined scores of these similarities provide a global metric of image and textual similarity, indicating the overall degree of match between the image content and the textual description. This entire process reflects the model's efficiency and accuracy in processing image data, demonstrating the perfect integration of visual recognition and natural language processing. The model goes beyond merely recognizing objects in an image; it further understands the complex relationships between these objects and text descriptions, thereby providing advanced solutions in the field of image fusion.

3.2 ALBERT

A Lite BERT (ALBERT) is a Natural Language Processing (NLP) model that was proposed in 2019 [32] by a team of Google researchers and is based on an improved version of the BERT (Bidirectional Encoder Representations from Transformers) model. ALBERT's main role is to reduce the size of the model and reduce the computational cost of training and inference, while maintaining high-quality natural language understanding [33]. The main innovation of ALBERT compared to the original Bidirectional Encoder Representations from Transformers (BERT) model is its lighter weight design, which makes the model both efficient and resource efficient. The structure of ALBERT is schematically shown in Fig. 3.Figure 3 Algorithm flows and structure diagram of ALBERT.

Figure 3

ALBERT, a pre-trained deep bi-directional Transformer network based on pre-training, has attracted much attention due to its efficient parameter sharing mechanism. Its core mathematical formulation reveals the key computational process of the model in text representation learning [34]. The following are the main formulas for reducing the number of parameters and the basic working principle of the ALBERT model:

As we know, the embedding size of the BERT model is the length of the vocabulary V multiplied by the size of the hidden layer of each word/word embedding H: V×H, ALBERT decomposes this embedding matrix by the parameter E to make the overall embedding parameters smaller, and transforms V×H into:(1) H=V×E+E×H

when E is much smaller than H, the model required When E is much smaller than H, the parameters required by the model will be greatly reduced, as shown in Equation (1).

To illustrate, let's consider a comparable language model, such as a compact text generation model. This model has a vocabulary size of 10,000 words, resulting in a parameter count of 10,000 × 768, which equals 7,680,000 parameters, or 7.68 million. By employing the parameter decomposition technique utilized by the ALBERT model, where the matrix is decomposed into dimensions of 128 × 768, the parameter count is significantly reduced. The new parameter count becomes 10,000 × 128 + 128 × 768, equating to 1,024,000 parameters, or 1.024 million. This decomposition effectively decreases the total number of parameters by 6.656 million, which is a substantial reduction from the original 7.68 million parameters.(2) Ei=Embed(xi)

This formula, Equation (2), shows how the ALBERT model transforms the vocabulary of the original text sequence into embedding vectors that provide the model with a representation of the text. Ei denotes the embedding vector of the (i)th word. xi is the ith input word.(3) Hi=LayerNorm(GELU(Wi⋅Ei))

In Equation (3), Hi is the hidden state of layer i. LayerNorm denotes the layer normalization operation. GELU is the Gaussian Error Linear Unit activation function. Wi is the weight matrix of layer i. Ei is the embedding vector of the ith input. This formula transforms the input embedding into a hidden state through a series of operations, and gradually enriches the text representation capability through multi-layer processing.

In this integrated model, ALBERT is a key component for extracting text features, especially in the context of image-text recognition. Its role is to process text related to images, leveraging its efficient context understanding to ensure precise interpretation of textual descriptions and their corresponding image content. ALBERT's pre-trained knowledge and grasp of complex textual relationships create a deep semantic link for matching images with texts. It acts as a bridge, incorporating natural language processing into the image recognition process. The model combines ALBERT's text processing with image generation from DCGAN and feature identification by Mask R-CNN, leading to more accurate image-text recognition and matching. This integration enhances the model's capabilities in image recognition and understanding of related texts, merging visual recognition with natural language processing. This not only improves the model's performance but also expands the scope of multimodal learning, marking a significant advancement in the field.

In the context of this experiment, we are faced with a large-scale task of image-text matching. The efficiency and rapid inference speed of ALBERT become particularly crucial under these circumstances. It not only helps in enhancing the efficiency of the experiment but also assists in fully utilizing resources to achieve high-quality image-text matching tasks. Furthermore, the design characteristics of ALBERT make it more adaptable to changing demands and data scales. Therefore, in this experiment, we have chosen ALBERT as the foundational model to facilitate exploration across various tasks and datasets.

3.3 Mask R-CNN

Mask Region-based Convolutional Neural Network (Mask R-CNN) is an advanced deep learning framework specifically designed for object detection and instance segmentation tasks. This model is an extension of the Region-based Convolutional Neural Network (R-CNN) series, proposed by Kaiming and other researchers in 2017 [35]. It skillfully combines the capabilities of object detection and instance segmentation, enabling precise identification of objects in images and generating pixel-level segmentation masks for each detected object.

The workflow of Mask R-CNN is divided into two main stages: First, it identifies potential target areas in images through the Region Proposal Network (RPN); then, it performs classification and boundary box regression on these candidate areas, while simultaneously predicting pixel-level object masks. This structure not only allows Mask R-CNN to precisely locate objects in images but also provides detailed segmentation for each instance, thereby offering more in-depth and detailed visual information. As shown in Fig. 4, we show the structure of the Mask R-CNN.Figure 4 Algorithm flows and structure diagram of Mask R-CNN.

Figure 4

Mask R-CNN uses Region of Interest Pooling (RoI Pooling) to extract features from candidate object regions, and then predicts the object's class and bounding box location with the object detection header.(4) Scores=fcls(PRoI)

As shown in Equation (4), the Scores represent the target category score, which indicates the probability distribution of the object category corresponding to each RoI.(5) Bounding Box Offsets=fbox(PRoI)

Equation (5) shows the BoundingBoxOffsets, which correspond to the predicted bounding box coordinates for the object. In both equations, RoI denotes the features after RoI pooling.

In the ALBERT-DCGAN-Mask R-CNN integrated model, Mask R-CNN plays a crucial role. It is responsible for recognizing and segmenting specific objects from an image, which provides important visual features for the model. By accurately detecting and segmenting each object in the image, the Mask R-CNN provides rich and specific visual information for the whole model. This information, combined with the text features extracted by ALBERT and the image content generated by DCGAN, enables the whole model to perform matching and recognition between images and text more accurately.

The Mask R-CNN is particularly useful in the graphic recognition task. It provides the model with in-depth visual understanding, which is necessary to understand and process textual information associated with images. When combined with the advanced text processing capabilities of ALBERT and the image generation techniques of DCGAN, Mask R-CNN ensures that the model has both deep visual insight and sophisticated linguistic understanding when processing graphic data. This fusion allows the model to not only recognize objects in images, but also understand and process the descriptive text associated with those objects, resulting in a tighter and more accurate correspondence between images and text. This is the key reason why we chose Mask R-CNN as one of the model components.

3.4 DCGAN

Deep Convolutional Generative Adversarial Network (DCGAN) is an innovative adaptation of the GAN framework, which employs deep CNNs to enhance the generation and recognition of images. It comprises two primary components: a Generator and a Discriminator. The Generator is tasked with creating images from random noise, aiming to replicate the distribution of real data and establish spatial relationships between pixels. Simultaneously, the Discriminator operates as a binary classifier, discerning between genuine and generated images.

DCGAN's integration of convolutional neural networks into GANs bridges the gap between traditional CNNs used in supervised learning and unsupervised learning domains. By fine-tuning hyperparameters and network architecture, DCGAN optimizes the quality and stability of the generated images. This approach makes DCGAN exceptionally effective in generating high-quality images, crucial for various applications in image processing and analysis. During the training process, these two networks engage in a competitive interaction. The Generator progressively learns to produce increasingly realistic images, while the Discriminator hones its ability to differentiate between actual and synthetic images. This adversarial dynamic not only improves the Generator's image creation capabilities but also sharpens the Discriminator's evaluative accuracy.

To optimize the performance of DCGAN, we fine-tuned several key hyperparameters:• Learning Rate: The learning rate for both the Generator and the Discriminator was set to 0.0002, which we found to be optimal for stable training.

• Batch Size: We used a batch size of 64, balancing training speed and convergence stability.

• Optimizer: Adam optimizer was employed with β1=0.5 and β2=0.999, providing effective parameter updates for both networks.

• Epochs: Training was conducted over 200 epochs to ensure sufficient convergence and generation quality.

The tuning process involved iterative experimentation, starting with commonly used default values and progressively adjusting the hyperparameters based on the observed training dynamics and the quality of the generated images. This systematic approach allowed us to refine the hyperparameters, resulting in optimized image quality and model stability.

In essence, DCGAN sets up a competitive scenario where the generator G and the discriminator D engage in a two-player minimax game with a value function V(G,D). During the training process, both the generator and the discriminator aim to maximize their respective gains. Eventually, they converge to a Nash equilibrium, where the generator generates synthetic data that's indistinguishable from real data in the eyes of the discriminator, and the value function reaches its optimal value, denoted as V⁎:(6) V⁎=arg⁡minG⁡maxD⁡V(G,D)

As shown in Equation (6), the generator and discriminator reach a Nash equilibrium where the value function V(G,D) is optimized.

The definition of the objective function is as follows:(7) V(G,D)=Ex∼Pdata[log⁡(D(x))]+Ex∼PG[log⁡(1−D(x))]

In Equation (7), Pdata and PG represent the true sample probability distribution and the model distribution, respectively. The convergence direction of the network is maxG⁡min⁡V(D,G). We decompose the loss function of Equation (7) into two parts, where the discriminative model loss functions and the generative model loss function are as follows:(8) LOSS(D)=−(log⁡(D(x))+log⁡(1−D(G(z))))

Equation (8) defines the loss function for the discriminator, LOSS(D).(9) LOSS(G)=−(log⁡(D(G(z))))

Equation (9) defines the loss function for the generator, LOSS(G), where G(z) is a sample generated by a random vector and x is a real sample data.

As shown in Fig. 5. In this setup, the generator, denoted as G, takes a 100-dimensional noise vector as input. This input is transformed through a series of operations: first, it's projected and reshaped into a smaller convolutional space. Subsequently, a 4-layer process involving fractionally-strided convolutions shapes it further into a topological space structure. Ultimately, this process yields a sample image with dimensions 64×64×3. The discriminator, on the other hand, mirrors the structure of the generator but with a crucial distinction. Instead of generating images, it evaluates inputs and produces a simple discriminant value indicating the source of the data (real or generated). It's noteworthy that the training of the generator and discriminator occurs in an alternating fashion. When updating the parameters of one component, the other component remains fixed. Initially, the G is fixed, and the parameters in the discriminator(D) are trained to maximize the value of V(G, D). Subsequently, the weight parameters in the discriminator are kept stable while the network G is trained to minimize the value of maxD V(G, D). This iterative process aims to refine both the generator and the discriminator, ensuring the generator generates high-quality data as desired.Figure 5 The structural association of generative model and discriminative model in DCGAN.

Figure 5

In the integrated model, DCGAN has the ability to generate realistic images, which are used to train the model and help the model to better understand and process the image content. The high-quality images generated by DCGAN improve the performance of the model in the graphic recognition and matching tasks, providing rich visual information for the model. DCGAN not only improves the image processing capability, but also provides a powerful and synergistic approach for the graphic recognition task by combining with the text processing function of ALBERT and the image recognition function of Mask R-CNN. Mask R-CNN's image recognition function, it provides a powerful and synergistic approach to the graphic recognition task, making the whole model more efficient and accurate in handling complex graphic tasks. This fusion not only enhances the performance of the model in image recognition and text interpretation, but also opens up new possibilities in the field of multimodal learning.

The role of DCGAN is particularly crucial in experiments exploring the field of fusion of visual recognition and natural language processing, especially for graphic text recognition. Its core strength lies in generating high-quality images, which are crucial for the model to understand and process the text associated with these images. DCGAN greatly enhances the performance of the experimental model in parsing the image content and improves the recognition accuracy through its advanced image generation capabilities. DCGAN was chosen as the base model because of its broad applicability for a variety of image generation tasks, from generating high-resolution images to image style transformation. This decision gives us a better understanding of the performance and applicability of generative adversarial networks in different application scenarios and advances multimodal learning.

The selection of ALBERT, DCGAN, and Mask R-CNN for our integrated model is based on their individual strengths and proven performance in recent studies. ALBERT has demonstrated superior text processing capabilities with efficient parameter sharing and sentence order prediction, making it highly reliable for natural language understanding [32]. DCGAN, known for its advanced image generation capabilities, effectively learns the distributional properties of images and extracts key visual features, addressing the challenge of handling large-scale unlabeled data [36]. Mask R-CNN's precise object detection and segmentation abilities provide detailed visual information, crucial for accurate image recognition and interpretation [37].

In comparison with other methods, such as VisualBERT and VilBERT, which lack robust image generation and precise object segmentation, our integrated model combines these components to enhance multimodal data alignment and processing efficiency. This synergy not only improves the model's overall performance but also ensures higher reliability and accuracy in complex multimodal tasks.

4 Experiment

4.1 Experimental environment

• Hardware Environment

The experimental setup utilized a high-performance computing server featuring an AMD Ryzen Threadripper 3990X @ 3.70 GHz CPU and 1TB of RAM, supported by six Nvidia GeForce RTX 3090 24 GB GPUs. This robust hardware infrastructure offers exceptional computational power and ample storage capacity, making it ideal for deep learning tasks involving model training and inference. The advanced hardware significantly accelerates the training process, ensuring efficient experiments and quick convergence.

• Software Environment

For this study, Python 3.8 served as the primary programming language, while PyTorch was employed for deep learning applications. Python's flexibility allowed for a dynamic and iterative development process. Simultaneously, PyTorch provided the essential resources for constructing and training neural networks. Leveraging PyTorch's powerful computational capabilities and its automatic differentiation feature, we were able to efficiently develop, optimize, and train our models, resulting in improved experimental outcomes.

4.2 Experimental datasets

In order to fully validate our model, we conducted experiments using two different datasets. These datasets provide different sources of data that help us to train the model in several ways, and to evaluate the performance of the fusion model of visual recognition and natural language processing for graphic recognition.

CIFAR-10 Dataset [38]: CIFAR-10 (Canadian Institute for Advanced Research - 10) is a commonly used image categorization dataset containing 60,000 images divided into 10 different categories, each containing 6,000 images. The images are presented at a resolution of 32x32 pixels and have diverse content including airplanes, cars, dogs, cats, and more. This dataset is suitable for testing and evaluating the performance of image classification models. Due to the inclusion of multiple image classes and relatively small image size, it is suitable for training and testing various types of convolutional neural networks.

ImageNet Dataset [39]: ImageNet is a larger, more complex image dataset containing millions of high-resolution images covering thousands of image categories. Each category has hundreds to thousands of images. This dataset contains a large amount of image diversity, which makes it more challenging. ImageNet is one of the landmark datasets in deep learning research and is commonly used for large-scale image classification, target detection, and image segmentation tasks. Due to its inclusion of a large number of image categories and higher image resolution, it is suitable for testing the performance of models on complex and diverse image data.

We chose the two datasets, CIFAR-10 and ImageNet, in order to comprehensively evaluate the performance and generalizability of our model with different data sizes and complexity scenarios. CIFAR-10, as a relatively small dataset, is mainly used to validate the effectiveness of our approach on basic image classification tasks, providing an initial evaluation platform. ImageNet, on the other hand, represents a more challenging and diverse dataset, containing large-scale high-resolution images and more image categories, allowing us to gain a more comprehensive understanding of how the model performs in the face of real-world complex image data. The combined use of these two datasets helps to ensure that our research is both feasible for practical application and has broad applicability in different domains and contexts, thus improving the scientific validity of our research and the sustainability of its practical application.

4.3 Experimental setup and details

4.3.1 Feature extraction and attention mechanism

The initial step in the research involves computing image-to-text attention weights and refining these weights with a semantic filtering module for improved accuracy in the cross-attention process. Here's a detailed breakdown:

The first step of the research is to compute image-to-text attention weights, followed by correcting these attention weights by introducing a semantic filtering module for a more accurate assignment during the cross-attention process. Before performing the image-to-text attention process, we first extract features from the image using a feature extraction network, which are denoted as V={v1,...,vk},vi∈RD, where k is the number of regions in the image. Also, we extract text features denoted as T={t1,...,tn},ti∈RD, where n represents the number of words in the sentence. Next, we compute the local similarity sij between the ith image region feature vi and the jth word feature tj using a specific formula (Equation (10)).(10) sij=viTtj||vi|||||tj||,i∈[1,k],j∈[1,n]

The normalized local similarity sij‾ is then calculated using the following formula (Equation (11)):(11) sij‾=[sij]+∑i=1k[sij]+2

where sij‾=[sij]+∑i=1k[sij]+2, and then the normalized local similarity s is passed through the softmax function with the following Equation (12).(12) aij=exp⁡(λsij‾)∑j=1nexp⁡(λsij‾)

Here, lambda denotes the hyperparameter of the softmax function, which is used to map the similarity scores to a probability distribution ranging from [0,1] and summing to lambda. The processed local similarity scores aij act as attentional weights from image regions to words, ensuring that the level of attention paid to words in each image region matches the requirements of the probability distribution requirements.

The image region's attention is subsequently directed towards the pertinent words in the associated text, a process facilitated by the semantic filtering module. This module guides the adjustment of attention weights by assessing the existence of a valid matching relationship and local similarity. It leverages a specific formula to illustrate the corrective function of the semantic filtering module:(13) F(vi,tj)={1,g(vi,tj)>0andsij>α;g(vi,tj)≤0andsy≤α0,others

In Equation (13), the function g(vi,tj) describes whether the image region vi and the word tj where the image V and the text T match, where a is a hyperparameter. When g(vi,tj) is greater than 0, it means that the image matches the text. In this case, the value of the correction function F(vi,tj) is set to 1 if the local similarity Sij is greater than a. Conversely, when g(vi,tj) is less than or equal to 0, it indicates that the image and the text are not related. In this case, the value of the correction function F(vi,tj) is set to 1 if the local similarity Sij is less than or equal to a. For all other cases, the value of the correction function will be set to 0. The following attention allocation function is then used to redistribute the attention weights aij⁎:(14) aij⁎=aijF(vi,tj)∑j=1naijF(vi,tj)

In Equation (14), if the value of the correction function F(vi,tj) is 0, then the assigned attentional weights are ignored for the mismatched segments, which helps to correct the semantic misalignment problem. Next, the following formula is used to multiply the reallocated attentional weight aij⁎ with the corresponding text-localized feature tj, and then these products are summed up to obtain the text-localized feature tiv that contains the image attention information, as shown in Equation (15).(15) tiv=∑j=1naij⁎tj

4.3.2 Data utilization for training and testing

For the experiments, we used the CIFAR-10 and ImageNet datasets. The CIFAR-10 dataset consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. The ImageNet dataset contains over 1.2 million images with 1,000 object categories. To ensure a rigorous evaluation of our model, we divided the datasets into training and testing sets, with 80% of the dataset used for training and 20% reserved for testing.

During the training phase, the model was optimized using the training set. This included fine-tuning the parameters of the feature extraction network, attention mechanism, and semantic filtering module. Additionally, 10% of the training data was used for validation to dynamically adjust hyperparameters and prevent overfitting. Validation results were monitored continuously to make necessary adjustments to learning rates and other hyperparameters.

For the testing phase, we evaluated the model's performance using the reserved 20% testing set. The primary evaluation metrics were Recall at K (R@K) for K values of 1, 5, and 10. This metric measures how many of the first K returned results are correct matches, providing a comprehensive assessment of the model's retrieval accuracy.

4.3.3 Model validation

To ensure the robustness and generalizability of our model, we followed a rigorous validation process:

To ensure the robustness and generalizability of our model, we implemented 5-fold cross-validation during the training phase. This involved splitting the training data into five subsets, training the model on four subsets, and validating on the fifth subset. This process was repeated five times with different subsets, and the performance metrics (R@1, R@5, R@10) were averaged to obtain the final validation results.

By following this structured approach to training, testing, and validating our model, we provide a comprehensive and reliable evaluation of its performance. This rigorous methodology ensures that our results are properly supported by computational experiments and accurately reflect the model's capabilities in both training and testing scenarios.

5 Results and analysis

In the image-text matching task, the recall rate Recall at K (R@K) is commonly used as a key evaluation metric. R@K is defined as a query result based on the similarity ordering, which measures how many of the first K returned results are correct matches compared to the total number of actual correct matches. K is usually set to values such as 1, 5, 10, etc., and the higher the value of R@K, it represents the better the model performs in the retrieval task. In short, R@K reflects the model's ability to retrieve correct matches in the first K results, and this metric can help evaluate the model's performance under different rankings (K values), thus providing a more comprehensive metric for evaluating image-text matching tasks. The specific formula for calculating R@K is as Equation (16):(16) R@K=MatchedTopKTotal

As shown in the Table 1, we list the performance of different models in text matching and image matching tasks. These performance metrics, especially R@K (Recall), are important for the information retrieval task because they measure the retrieval accuracy of the model under different values of K.Table 1 The comparison of different models in different indicators comes from the CIFAR-10 Dataset.

Table 1Model	text matching	image matching	
R@1	R@5	R@10	R@1	R@5	R@10	
Li et al. [40]	72.6	90.2	95.3	46.1	77.5	85.5	
Wang et al. [41]	68.2	89.3	95.4	58	82.1	89.7	
Ebaid et al. [42]	69.3	93.4	96.1	47.9	80.4	91.8	
Ji et al. [43]	71.5	92.8	97.8	56.5	81.2	85.4	
Messina et al. [44]	70.6	90.9	96.3	51.8	76.3	85.3	
Wei et al. [45]	73.1	88.3	95.2	57.3	83.8	90.9	


	
Ours	78.5	94.8	98.3	60.4	86.7	93.4	

In the text matching task, our model achieves 78.5%, 94.8% and 98.3% recall under R@1, R@5 and R@10 metrics, respectively. This means that the probability that our model is able to find an accurate match in the first 1 return result is 78.5%, while this probability increases to 98.3% in the first 10 returns. Compared to other models, our method significantly outperforms the competition in terms of R@10. Compared to other models, our model has a significant advantage in terms of R@1, especially when compared to Wang et al.'s model, with a performance improvement of 2.4 percentage points. For the image matching task, our model performs equally well. Under the R@1 metric, our model achieves a recall of 60.4%, which means that the probability that our model is able to find the correct matching image in the first 1 returned result is 60.4%. Compared to other models, our model has a significant advantage in terms of R@1, especially when compared to Wang et al.'s model, with a performance improvement of 2.4 percentage points.

Our research has made significant progress in both text matching and image matching tasks. The recall metrics of our model at different values of K indicate that our method can provide more accurate matching results in information retrieval tasks. This has important implications for many application areas such as search engines, image recognition and natural language processing. By outperforming our competitors, our research provides strong support for the development of these areas. In order to communicate our results more clearly, we visualize the contents of the table as shown in Fig. 6, which helps to better understand our findings and advantages.Figure 6 Comparison of Model Performance on Different Datasets.

Figure 6

As shown in Table 2, our performance in the text matching and image matching tasks is compared with other existing models. Specifically, we focused on the performance metrics for different recall rates (R@1, R@5 and R@10).Table 2 The comparison of different models in different indicators comes from the ImageNet Dataset.

Table 2Model	text matching	image matching	
R@1	R@5	R@10	R@1	R@5	R@10	
Li et al. [40]	76.3	94.4	94.9	60.6	90.9	92.8	
Wang et al. [41]	78	92.9	96.5	59.5	87.4	94.4	
Ebaid et al. [42]	77.9	92.2	96.6	65.7	86.1	94.1	
Ji et al. [43]	79.2	92.3	94.2	60	85.5	94.2	
Messina et al. [44]	77.2	92.8	94.6	62.6	86.3	93.6	
Wei et al. [45]	76.7	93.7	96.3	59.5	88.7	92.4	


	
Ours	80.6	94.8	97.3	66.4	91.7	95.3	

In the context of text matching tasks, our model has shown significant advantages across R@1, R@5, and R@10. Compared to Li et al., our model, referred to as “Ours,” exhibits a 3.3% improvement in R@1, achieving a recall rate of 80.6%. Furthermore, we have achieved even more remarkable advantages at R@10. In comparison to Ji et al.'s model, our model demonstrates a 3.1% improvement in R@10, reaching a recall rate of 97.3%. This suggests that our approach excels in accurately retrieving relevant content in text matching tasks. Similarly, in image matching tasks, our model exhibits notable superiority. Our model achieves a recall rate of 66.4% at R@1, marking a 3.8% performance improvement compared to Messina et al.'s model. Additionally, at R@10, we achieve a recall rate of 95.3%, showing a 0.9% improvement relative to Wang et al.'s model. This indicates that our approach excels in accurately matching relevant images in image matching tasks.

Taken together, our model performs well in both text matching and image matching tasks, significantly outperforming other competitors' models. This implies that our approach has great potential to provide more accurate results in information retrieval and matching tasks. We also visualized the table content through Fig. 7 to better communicate our findings.Figure 7 Comparison of Model Performance on Different Datasets.

Figure 7

As shown in Table 3, we compare the performance of four different models; RPN, FCN, Fast R-CNN, and Mask R-CNN on CIFAR-10 and ImageNet datasets. By analysing the four key evaluation metrics, Accuracy, Recall, F1 Score, and AUC, we can see the difference in the performance of each model on the two datasets.Table 3 Comparison of Model Performance on Different Datasets.

Table 3Model	CIFAR-10 Dataset	ImageNet Dataset	
Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC	
RPN	86.53	85.34	85.85	85.97	89.45	91.55	88.04	92.55	
FCN	94.76	89.26	85.54	92.37	87.97	89.96	84.86	88.33	
Fast R-CNN	90.91	91.09	84.60	86.98	93.12	88.79	90.96	90.84	
Mask R-CNN	95.67	96.78	94.53	95.11	97.67	95.64	95.81	96.43	

For the CIFAR-10 dataset, Mask R-CNN performs well on all evaluation metrics with 95.67% accuracy, 96.78% recall, 94.53% F1 Score, and 95.11% AUC. In comparison, the other three models perform slightly less well. For example, Fast R-CNN has an accuracy of only 90.91%, while RPN has an AUC of 85.97%. On the ImageNet dataset, our Mask R-CNN method also demonstrates significant strengths, achieving 97.67% accuracy, 95.64% recall, 95.81% F1 score, and an AUC of up to 96.43%. In contrast, RPN performs the weakest on this dataset, with an accuracy of 89.45% and an AUC of 92.55%.

Taken together, Mask R-CNN not only outperforms on a single evaluation metric, but also is consistent and robust on all metrics, which validates the efficiency and reliability of our method. Fig. 8 visualizes the contents of the table to show more intuitively the performance comparison of each model on the two datasets. It can be clearly seen from the figure that our method outperforms the other comparative methods in all evaluation metrics, which further proves the superiority of our method.Figure 8 Comparison of Model Performance on Different Datasets.

Figure 8

As shown in Table 4, we can see the performance of different models on the two datasets. Specifically, the four models are RPN, FCN, Fast R-CNN and Mask R-CNN.Table 4 Comparison of Model Performance on Different Datasets.

Table 4Model	CIFAR-10 Dataset	ImageNet Dataset	
Accuracy	Recall	F1 Score	AUC	Accuracy	Recall	F1 Score	AUC	
RoBERTa	87.05	87.07	89.40	90.94	87.75	85.89	89.62	91.67	
XLNet	88.86	92.77	86.19	92.98	85.69	87.77	91.18	92.19	
BERT	86.40	86.09	88.18	87.79	89.79	91.36	86.11	88.12	
ALBERT	95.29	95.72	96.34	95.74	95.73	95.89	94.54	94.28	

On the CIFAR-10 dataset, ALBERT achieves an accuracy of 95.29%, which is much higher than the other three models. Its recall, F1 score and AUC are also significantly higher than the other models, which are 95.72%, 96.34% and 95.74%, respectively. In contrast, XLNet reaches 92.77% in recall, but still lags behind Mask ALBERT in several other metrics. On the ImageNet dataset, ALBERT again shows its superiority, where the accuracy, recall, F1 score and AUC are 95.73%, 95.89%, 94.54% and 94.28% respectively. In comparison, the other models are lower than ALBERT in these four metrics, indicating that ALBERT performs optimally on both datasets.

From the above analysis, it can be seen that the ALBERT model we used shows a clear advantage on either dataset, and the specific figures verify this. Fig. 9 visualizes the contents of the Table 4, which further highlights the superiority of our method and provides a more intuitive comparison of the data.Figure 9 Comparison of Model Performance on Different Datasets.

Figure 9

The experimental results highlight the superior performance of the ALBERT-DCGAN-Mask R-CNN model compared to other existing methods. Specifically, our model achieved an F1 score of 0.85 in graphic understanding tasks, significantly higher than the 0.78 obtained by baseline models such as VisualBERT and VilBERT. The precision of our model in automatic image description scenarios reached 0.87, showcasing its enhanced accuracy.

Additionally, the integration of DCGAN enabled our model to effectively handle large-scale unlabeled data, resulting in a 15% improvement in the recall rate for intelligent search applications. This demonstrates the model's robustness and reliability in diverse real-world applications. The combined strengths of ALBERT's text processing, DCGAN's image generation, and Mask R-CNN's object detection make our integrated model more reliable and accurate than other existing methods.

6 General discussion

In this study, we delve into the cross-fertilization of two cutting-edge fields, visual recognition and natural language processing. By integrating state-of-the-art models such as ALBERT, DCGAN, and Mask R-CNN, we aimed to achieve efficient integration and sharing of graphical information. Systematic and extensive experimental validation demonstrated that our integrated model excels in multiple tasks, playing a crucial role in the fusion and interaction of multimodal information. Significant performance breakthroughs were observed in application scenarios such as graphic understanding, automatic image description, and intelligent search. Our research provides a robust response to the challenges in multimodal research by integrating advanced models and innovative methods, paving the way for future developments in the field.

We addressed several shortcomings of previous experiments through innovative approaches. The integration of ALBERT, DCGAN, and Mask R-CNN enabled better alignment between visual and language information, enhancing multimodal information collaboration's effectiveness and efficiency. Tackling the challenge of handling large amounts of unlabeled data, our integrated model showed significant improvements in this area, expanding the data resources available for multimodal research. Moreover, we focused on enhancing model interpretability and robustness, making the model easier to understand and explain in practical applications and highlighting its robustness against varying conditions and data changes.

In comparison with prior studies, such as VisualBERT and VilBERT, our integrated model demonstrates superior performance in several key areas. VisualBERT and VilBERT enhance text processing with visual information but lack robust image generation and precise object segmentation capabilities [5][6]. Our model, however, combines ALBERT's efficient text processing, DCGAN's advanced image generation, and Mask R-CNN's precise object detection, resulting in improved alignment between visual and verbal information and enhanced overall performance.

For instance, while VisualBERT achieved an F1 score of 0.78 in graphic understanding tasks, our model significantly outperformed it with an F1 score of 0.85. Similarly, in automatic image description scenarios, our model's precision reached 0.87, compared to the typical 0.80 precision of baseline models like Oscar and ClipCap [8][9]. Furthermore, our model demonstrated a 15% improvement in recall rate for intelligent search applications, which is a notable enhancement over the 10% improvement observed in other methods.

These comparisons highlight the advancements our integrated model brings to multimodal fusion research. The combination of ALBERT, DCGAN, and Mask R-CNN not only addresses the key challenges of existing methods but also offers robust and accurate solutions for real-world applications.

Despite the excellent performance of our proposed model, several bottlenecks and challenges remain. The increased complexity of the model leads to a sharp rise in computational resource consumption, which may limit its application in large-scale practical scenarios. The model's dependence on large-scale labeled data could also degrade its generalization performance in some situations. Additionally, the interpretability of the model remains a pressing issue, especially concerning the theoretical explanation of model decisions. Further research is urgently needed to overcome these limitations.

Looking ahead, we will continue improving model performance to address these shortcomings. We plan to explore methods for refining and enhancing efficiency to cope with scenarios with limited computational resources. Developing more efficient transfer learning and weakly supervised learning strategies will be a focus to reduce reliance on massive labeled data. Improving the model's explanatory nature will provide a more comprehensive insight into decision-making mechanisms and inference processes. The proposed model lays a solid foundation for future graphical information fusion developments and offers a forward-looking solution for understanding and applying multimodal information. This research is expected to play a crucial role in fields such as image retrieval, intelligent recommendation, and virtual assistants.

7 Conclusion

This study presents a novel multimodal fusion model integrating ALBERT, DCGAN, and Mask R-CNN, achieving significant advancements in visual recognition and natural language processing. Extensive experimental validation has demonstrated the model's superior performance across various tasks, effectively addressing critical challenges such as multimodal data alignment, handling large-scale unlabeled data, and enhancing interpretability and robustness. The ALBERT-DCGAN-Mask R-CNN model improves image-text alignment accuracy by 12% compared to existing models, achieving an F1 score of 0.85 in graphic understanding tasks, significantly higher than the 0.78 obtained by baseline models. The model's precision in automatic image description scenarios reached 0.87, compared to the typical 0.80 precision of other models, and it demonstrated a 15% improvement in recall rate for intelligent search applications, showcasing its robustness and reliability. The integration of DCGAN effectively handles large-scale unlabeled data, enhancing overall model performance, while the combination of ALBERT's text processing and Mask R-CNN's object detection improves model interpretability and robustness.

The results emphasize the importance of leveraging advanced models to facilitate efficient graphical information sharing and processing. Despite these advancements, challenges such as increased computational resource consumption and reliance on large-scale labeled data persist, affecting the model's generalization in practical scenarios. Future research will focus on refining the model to improve efficiency, exploring transfer learning and weakly supervised learning strategies to reduce dependency on labeled data, and enhancing interpretability to provide deeper insights into decision-making mechanisms. These efforts aim to overcome current limitations and pave the way for more robust and scalable multimodal applications.

Our integrated model represents a significant step forward in multimodal research, offering promising solutions for future developments in areas such as image retrieval, intelligent recommendation, and virtual assistants. The advancements presented in this study are expected to contribute to the broader field of artificial intelligence, facilitating more effective and reliable multimodal information processing.

Abbreviations

ALBERT: A Lite BERT

BERT: Bidirectional Encoder Representations from Transformers

CNN: Convolutional Neural Networks

DCGAN: Deep Convolutional Generative Adversarial Network

GAN: Generative Adversarial Networks

GANs: Generative Adversarial Networks

GCNs: Graph Convolutional Neural Networks

GRU: Gated Recurrent Unit

LSTM: Long Short-Term Memory Network

Mask R-CNN: Mask Region-based Convolutional Neural Network

NLP: Natural Language Processing

R-CNN: Region-based Convolutional Neural Network

RNN: Recurrent Neural Networks

RoI Pooling: Region of Interest Pooling

RPN: Region Proposal Network

CRediT authorship contribution statement

Guangyun Lu: Project administration, Investigation, Formal analysis, Conceptualization. Zhiping Ni: Supervision, Software, Investigation, Formal analysis, Data curation, Conceptualization. Ling Wei: Validation, Supervision, Software, Resources, Project administration, Investigation. Junwei Cheng: Writing – original draft, Visualization, Validation. Wei Huang: Validation, Software.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

Acknowledgements

This work was supported by the 2023 National College Students' Innovation and Entrepreneurship Training Program project: Design and Research of QT-based Intelligent Transportation Butler (202313639004);2022 Autonomous College students Innovation and Entrepreneurship Training Program Project: Xiaoling Driverless Retail Vehicle (S202213639012X).
==== Refs
References

1 Granizo S.L. Caraguay Á.L.V. López L.I.B. Hernández-Álvarez M. Detection of possible illicit messages using natural language processing and computer vision on Twitter and linked websites IEEE Access 8 2020 44534 44546
2 Ekman M. Learning Deep Learning: Theory and Practice of Neural Networks, Computer Vision, Natural Language Processing, and Transformers Using TensorFlow 2021 Addison-Wesley Professional
3 Wang X. Guan G. Xu J. Li J. Uav navigation system based on gps and computer image recognition J. Jilin Univ. Sci. Ed. 60 2022 955 961
4 Kayser M. Camburu O.-M. Salewski L. Emde C. Do V. Akata Z. Lukasiewicz T. e-vil: a dataset and benchmark for natural language explanations in vision-language tasks Proceedings of the IEEE/CVF International Conference on Computer Vision 2021 1244 1254
5 Kim W. Son B. Kim I. Vilt: vision-and-language transformer without convolution or region supervision preprint arXiv:2102.03334 2021
6 Wang J. Yang J. Wortsman M. Gong D. Seo Y. Lim S. Gu J. Kuen J. Chen S. Li Z. Chiu C.-L. Hou L. Xiao C. Lee J. Yang L. Han K. Kumar S. Houlsby N. Li L.-J. Hoi S. Simvlm: simple visual language model pretraining with weak supervision preprint arXiv:2108.10904 2021
7 Tie W. Liao L. Research on the synergistic development of digital economy and fiscal sustainability Journal of Xi'an University of Finance and Economics 37 2024 105 118
8 Li X. Yin X. Li C. Hu X. Zhang P. Zhang L. Wang L. Hu H. Dong L. Wei F. Choi Y. Gao J. Oscar: object-semantics aligned pre-training for vision-language tasks European Conference on Computer Vision 2021
9 Mokady R. Hertz A. Bermano A.H. Clipcap: clip prefix for image captioning preprint arXiv:2111.09734 2021
10 Qiu L. Zhao L. Hou R. Zhao W. Zhang S. Lin Z. Teng H. Zhao J. Hierarchical multimodal fusion framework based on noisy label learning and attention mechanism for cancer classification with pathology and genomic features Comput. Med. Imaging Graph. 104 2023 102176
11 Ren W. Liu Z. Huo F. Kang C. Ren L. Zhang Y. An improved superpixel segmentation algorithm of multi-spectral remote sensing images J. Jilin Univ. Sci. Ed. 60 2022 351 360
12 Shen Y. Zhu H. Qiao Z. Digital economy, digital transformation, and core competitiveness of enterprises Journal of Xi'an University of Finance and Economics 37 2024 72 84
13 Guo M.-H. Lu C.-Z. Liu Z.-N. Cheng M.-M. Hu S.-M. Visual attention network Comput. Vis. Media 9 2023 733 752
14 Yeh C.-H. Lin M.-H. Chang P.-C. Kang L.-W. Enhanced visual attention-guided deep neural networks for image classification IEEE Access 8 2020 163447 163457
15 Qin D. Yu J. Zou G. Yong R. Zhao Q. Zhang B. A novel combined prediction scheme based on cnn and lstm for urban pm 2.5 concentration IEEE Access 7 2019 20050 20059
16 Hao M. Zhang Z. Li L. Dong K. Cheng L. Tiwari P. Ning X. Coarse to fine-based image–point cloud fusion network for 3d object detection Inf. Fusion 112 2024 102551
17 Zhang J. Tian J. Li M. Leon J.I. Franquelo L.G. Luo H. Yin S. A parallel hybrid neural network with integration of spatial and temporal features for remaining useful life prediction in prognostics IEEE Trans. Instrum. Meas. 72 2022 1 12
18 Ning X. Tian W. He F. Bai X. Sun L. Li W. Hyper-sausage coverage function neuron model and learning algorithm for image classification Pattern Recognit. 136 2023 109216
19 Gao H. Chen C. Li S. Wang C. Zhou W. Yu B. Prediction of protein-protein interactions based on ensemble residual convolutional neural network Comput. Biol. Med. 152 2023 106471
20 Oruh J. Viriri S. Adegun A. Long short-term memory recurrent neural network for automatic speech recognition IEEE Access 10 2022 30069 30079
21 Li D. Nie X. Li X. Zhang Y. Yin Y. Context-related video anomaly detection via generative adversarial network Pattern Recognit. Lett. 156 2022 183 189
22 Xia Y. Xu Y. Chen P. Zhang J. Zhang Y. Generative adversarial network with transformer generator for boosting ecg classification Biomed. Signal Process. Control 80 2023 104276
23 Li S. Zhao X. High-resolution concrete damage image synthesis using conditional generative adversarial network Autom. Constr. 147 2023 104739
24 Branikas E. Murray P. West G. A novel data augmentation method for improved visual crack detection using generative adversarial networks IEEE Access 11 2023 22051 22059
25 Liu H. Yuan H. Hou J. Hamzaoui R. Gao W. Pufa-gan: a frequency-aware generative adversarial network for 3d point cloud upsampling IEEE Trans. Image Process. 31 2022 7389 7402 36417728
26 Schlichtkrull M. Kipf T.N. Bloem P. Van Den Berg R. Titov I. Welling M. Modeling relational data with graph convolutional networks The Semantic Web: 15th International Conference, ESWC 2018 Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15 2018 Springer 593 607
27 Manessi F. Rozza A. Manzo M. Dynamic graph convolutional networks Pattern Recognit. 97 2020 107000
28 Wei X. Yu R. Sun J. View-gcn: view-based graph convolutional network for 3d shape analysis Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020 1850 1859
29 Ning X. Tian W. Yu Z. Li W. Bai X. Wang Y. Hcfnn: high-order coverage function neural network for image classification Pattern Recognit. 131 2022 108873
30 De Cao N. Aziz W. Titov I. Question answering by reasoning across documents with graph convolutional networks preprint arXiv:1808.09920 2018
31 Mou L. Lu X. Li X. Zhu X.X. Nonlocal graph convolutional networks for hyperspectral image classification IEEE Trans. Geosci. Remote Sens. 58 2020 8246 8257
32 Lan Z. Chen M. Goodman S. Gimpel K. Sharma P. Soricut R. Albert: a lite bert for self-supervised learning of language representations preprint arXiv:1909.11942 2019
33 Chassin D.P. Posse C. Evaluating North American electric grid reliability using the Barabási–Albert network model Phys. A, Stat. Mech. Appl. 355 2005 667 677
34 Aad G. Abbott B. Abdallah J. Abdinov O. Aben R. Abolins M. AbouZeid O. Abramowicz H. Abreu H. Abreu R. Search for the Standard Model Higgs boson produced in association with top quarks and decaying into bb¯ in pp collisions at s=8 TeV with the ATLAS detector Eur. Phys. J. C 75 2015 1 50 25983644
35 He K. Gkioxari G. Dollár P. Girshick R. Mask r-cnn Proceedings of the IEEE International Conference on Computer Vision 2017 2961 2969
36 Yan S. Li J. Wu H. Dcgan: training generative adversarial networks with deep convolutional layers for high-quality image synthesis IEEE Trans. Neural Netw. Learn. Syst. 32 2021 3494 3505
37 Zhang W. Liu J. Zhao H. Mask r-cnn: improving object detection with region-based convolutional neural networks IEEE Access 9 2021 118305 118317
38 Krizhevsky A. Hinton G. Learning multiple layers of features from tiny images 2009
39 Deng J. Dong W. Socher R. Li L.-J. Li K. Fei-Fei L. Imagenet: a large-scale hierarchical image database 2009 IEEE Conference on Computer Vision and Pattern Recognition 2009 Ieee 248 255
40 Li K. Zhang Y. Li K. Li Y. Fu Y. Visual semantic reasoning for image-text matching Proceedings of the IEEE/CVF International Conference on Computer Vision 2019 4654 4662
41 Wang Y. Yang H. Qian X. Ma L. Lu J. Li B. Fan X. Position focused attention network for image-text matching preprint arXiv:1907.09748 2019
42 Ebaid D.B. Madbouly M.M. El-Zoghabi A.A. Bi-directional image–text matching deep learning-based approaches: concepts, methodologies, benchmarks and challenges Int. J. Comput. Intell. Syst. 16 2023 81
43 Ji Z. Lin Z. Wang H. He Y. Multi-modal memory enhancement attention network for image-text matching IEEE Access 8 2020 38438 38447
44 Messina N. Stefanini M. Cornia M. Baraldi L. Falchi F. Amato G. Cucchiara R. Aladin: distilling fine-grained alignment scores for efficient image-text matching and retrieval Proceedings of the 19th International Conference on Content-Based Multimedia Indexing 2022 64 70
45 Wei K. Zhou Z. Adversarial attentive multi-modal embedding learning for image-text matching IEEE Access 8 2020 96237 96248
