
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)12696-5
10.1016/j.heliyon.2024.e36665
e36665
Research Article
Refining CycleGAN with attention mechanisms and age-Aware training for realistic Deepfakes
Cheng Xi 20030032@sasu.edu.cn

School of Health Caring Industry, Sichuan University of Arts and Science, Dazhou, Sichuan, 635000, China
22 8 2024
30 8 2024
22 8 2024
10 16 e3666526 5 2024
17 8 2024
20 8 2024
© 2024 The Author
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
In the evolving landscape of deep learning technologies, the emergence of Deepfakes and synthetic media is becoming increasingly prominent within digital media production. This research addresses the limitations inherent in existing face image generation algorithms based on Generative Adversarial Networks (GAN), particularly the challenges of domain irrelevancy and inadequate facial detail representation. The study introduces an enhanced face image generation algorithm, aiming to refine the CycleGAN framework. The enhancement involves a two-fold strategy: firstly, the generator's architecture is refined through the integration of an attention mechanism and adaptive residual blocks, enabling the extraction of more nuanced facial features. Secondly, the discriminator's accuracy in distinguishing real from synthetic images is improved by incorporating a relative loss concept into the loss function. Additionally, this study presents a novel model training approach that incorporates age constraints, thereby mitigating the effects of age variations on the synthesized images. The effectiveness of the proposed algorithm is empirically validated through comparative analysis with existing methodologies, utilizing the CelebA dataset. The results demonstrate that the proposed algorithm significantly enhances the realism of generated face images, outperforming current methods in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), while also achieving notable improvements in subjective visual quality. The implementation of this advanced method is anticipated to substantially elevate the efficiency and quality of digital media production, contributing positively to the broader field of digital media creation.

Keywords

Deepfakes
Generative adversarial networks
Synthetic media
Multimedia
Digital media production
==== Body
pmc1 Introduction

Within the dynamic sphere of digital media technology, an ascending demand for sophisticated digital media production is discernible. The principal objective of such production lies in delivering a superior visual experience and augmenting user immersion, as noted by Ref. [1]. Traditional methodologies in digital media production, however, encounter considerable obstacles in managing intricate scenes and actualizing high-grade visual effects. The integration of deep learning techniques has catalyzed significant breakthroughs in this domain. Particularly, in digital media, the advent of Deepfakes and synthetic media has marked a pivotal research and development frontier, with applications spanning image composition, video editing, and facial synthesis [[2], [3], [4]]. These techniques, underpinned by deep learning, are capable of autonomously discerning key data features and patterns, subsequently enabling the generation of synthetic images and videos of high fidelity.

Recent research endeavors have underscored the increasing utilization of Deepfake and synthetic media technologies within digital media, attracting substantial academic attention. Various deep learning-based algorithms for Deepfakes and synthetic media have been proposed. In the context of image synthesis [5], introduced a GAN-based algorithm capable of producing realistic, high-resolution images, albeit with complexities like hyperparameter adjustments and potential training instability [6]. developed a cGAN-based image conversion algorithm effective in style transfer, yet requiring specific tuning for diverse datasets and tasks. To address these challenges, the incorporation of the self-attention mechanism has been explored for stabilizing and enhancing image quality, as demonstrated in Ref. [7] with the introduction of MaskGAN. Despite its complexity, MaskGAN represents a multi-task learning framework, amalgamating generation and recognition tasks.

In video editing, advancements have been made through deep learning-based algorithms. For instance Ref. [8], presented an algorithm for transferring facial expressions between individuals in videos, although its efficacy is contingent on the quality of input data and faces challenges in complex scenarios. This algorithm also facilitates face reconstruction and replacement. Multi-view video synthesis, as discussed in Ref. [9], enhances the realism of 3D scene reconstruction by integrating videos from multiple viewpoints. Additionally, applications of these technologies in emerging areas like augmented reality are being explored for more immersive experiences, as indicated in Ref. [10].

Regarding face synthesis, significant strides have been made, as evidenced by Ref. [11], which introduced a cGAN-based technique for converting non-frontal to frontal face images, achieving high-resolution synthesis. Despite the broad application potential of Deepfake and synthetic media in digital media production, current techniques exhibit limitations such as a lack of diversity in generated images, temporal incoherence in videos, and insufficient facial detail.

To mitigate these shortcomings, this research proposes an enhanced CycleGAN-based algorithm for face image generation. The proposed method includes optimizing the generator within the CycleGAN framework by integrating an attention mechanism and adaptive residual blocks, thereby refining the image transformation process and image quality. Additionally, the discriminator's effectiveness is enhanced through an improved loss function, addressing issues of discrimination accuracy. Moreover, a novel age constraint-based model training strategy is introduced to minimize the impact of age variation on generated images.

This research is organized as follows: Section I introduces the study, outlining its context, significance, current state, and objectives. Section II delves into the historical background and developments related to deep forgery and synthetic media. Section III details the proposed algorithmic model and its optimization. Section IV presents the experimental design, results, and comparative analysis. Finally, Section V encapsulates the research contributions and innovations.

2 Related works

2.1 Face feature generation

Face feature generation is like appearance style transfer methods [12] in that both require variations to generate local key features while preserving the original identity information as much as possible. Traditional face feature generation methods use image processing techniques [13] to generate face pictures by extracting the positions of the five features of a face and calling the corresponding textures from a library to match or replace them according to the desired feature style. In the field of video image processing, the artistic features of the face region are simulated by learning specific styles of stroke features using realistic rendering [14]. In the literature [15], a variational autoencoder was proposed to adjust the encoder by applying a priori to potential cells in order to sample or insert into potential cells to complete the face image. Despite its effectiveness, the training objective of the variational autoencoder is rooted in the Gaussian likelihood function at the pixel level. As a result, the images synthesized through this approach often appear excessively smooth or even blurred. This is advantageous in restoring smoother objects such as blue sky and white clouds, but is not ideal for face images. Therefore, the literature [16] improves the variational autoencoder by adding a discriminator derived from GAN and proves that it can generate more realistic face images. Training the variational autoencoder and the adversarial loss function together prevents over-smoothing but may lead to distortion. This approach requires significant time and labor costs for the initial sampling effort, which is difficult to perform smoothly in practical applications.

Currently, generative adversarial networks are being used more often to generate more realistic face images. Generative adversarial networks have been proven by many researchers to generate high quality images [17] and have achieved good results in numerous other aspects such as text generation. In general, GANs use the idea of dynamic gaming to synthesize what it considers to be real images through constant dynamic adversarial between generators and discriminators. However, most GAN-based generation methods are limited by the number and quality of original image samples, which makes it difficult to guarantee the clarity of the result and the preservation of identity features. So, people further considered introducing diverse attention mechanisms in generative adversarial networks to improve the network's attention to local relevance and better focus on the generated features [18]. Efforts have also been made to address the issue of poor stability in generative adversarial networks by establishing gradient propagation through multi-scale feature maps. This approach improves the model's overall performance in dealing with various types of datasets of differing sizes and image quality. Contrast learning methods are now also introduced into generative adversarial networks to ensure that the models can have more diverse generative results based on good quality of the generated images.

At the same time, the evaluation of the results of such tasks should not only compare data differences and changes in the results of mathematical analysis, but also consider the subjective visual perception of the actual experimenter. The subjective evaluation method based on experimenter participation is a relatively objective evaluation method that combines the above two points. The experimenter should rate the generated face images according to the target task for evaluation.

2.2 Generating adversarial networks

Generative adversarial networks have gained significant traction in face image synthesis [19]. There are numerous image generation models, but not all of them are suitable for small-scale data generation containing noisy interference. Among them, generative adversarial networks balance ease of use with extensibility, such as the use of conditional generative networks to solve the image-to-image conversion task in the pix2pix [20] model, which extends to the high-resolution level in the pix2pixHD model. The StyleGAN [21] model proposes a novel generator architecture to synthesize images with excellent results, which controls the generator by adaptive instance normalization. Harms et al. [22] proposed an unsupervised method without paired data, Cycle Generative Adversarial Network (CycleGAN), to generate images using cyclic consistency. Taking CycleGAN as an example, which uses a ring network structure including two generators and two discriminators, the model structure of CycleGAN is shown in Fig. 1.Fig. 1 Schematic diagram of CycleGAN.

Fig. 1

CycleGAN is a type of unsupervised generative adversarial network that implements image style transformation between two domains using a symmetric generator ring structure. Unlike previous generative adversarial networks, CycleGAN have only one set of generator-discriminator pairs. CycleGAN uses a symmetric generator ring structure to perform better cyclic adversarial training. To improve the generators in the CycleGAN network structure, this study presents an attention mechanism and adaptive residual blocks. This approach enables the model to learn and extract facial features more efficiently during image transformation, resulting in higher-quality generated images.

3 Methodology

3.1 The proposed architecture

This research proposes a model that adopts the CycleGAN network structure with two sets of generators and discriminators. The architecture of the model, as illustrated in Fig. 2, comprises two generative adversarial networks. Specifically, the image domain X is transformed into the forged image domain YF through the generator G. discriminator DY discriminates whether the forged image belongs to image domain Y. The forged image domain YF passes through generator F to get the reconstructed image domain XR. The loop consistency loss between image domain X and reconstructed image domain XR ensures the validity of image translation. It guides the mapping of image domain X to image domain Y and reduces the influence of irrelevant image domains on the image translation. Similarly, the image domain Y passes through the generator F to get the forged image domain XF and then passes through the generator G to get the reconstructed image domain YR.Fig. 2 The proposed architecture.

Fig. 2

During the training phase of the model, the generative network is used to produce forged and reconstructed images for the input image domains X and Y, respectively. During the model testing phase, the most recent saved model is loaded. Next, input images are fed into the generative network to obtain the corresponding forged and reconstructed images. Finally, the resulting forged and reconstructed images are saved.

3.2 Generator structure

To enhance the quality and naturalness of the generated images, this research proposes the use of an attention mechanism and an adaptive residual block with adaptive layer instance normalization. Fig. 3 shows the improvement of generator structure. The generator is divided into three parts: the down sampling region, the middle region with the adaptive residual block and the normal residual block, and the up-sampling region with the attention mechanism.Fig. 3 The improvement of generator structure.

Fig. 3

3.2.1 Attention mechanism

The attention mechanism mimics human observation by emphasizing important local features in images. It has been widely used in various computer vision tasks, including machine translation, target detection, and image classification. There are two main types of attention mechanisms: soft attention, which learns attention weights using neural networks, and hard attention, which focuses on specific location information through augmented learning. In this study, the authors utilized a convolutional attention mechanism that combines both channel and spatial dimensions. It involves computing a one-dimensional channel attention map and a two-dimensional spatial attention map. For a feature map F with dimensions F∈RC×B×M, the computation of the attention maps is performed independently along the channel and spatial dimensions, as shown in Equations (1), (2).(1) F′=Wc(F)⊗F

(2) F″=Ws(F′)⊗F′

In the equations above, the symbol ⊗ represents element-wise multiplication. The convolutional attention mechanism used in this research computes the channel attention map Wc∈RC×1×1 and the spatial attention map Ws∈R1×B×M separately along the channel and spatial dimensions of the given feature map respectively. Compared with the attention mechanism used in SEnet [23], which only focuses on the channel dimension, the convolutional attention mechanism used in this research is more effective. Fig. 4 shows the channel attention module used in this mechanism. Unlike general attention mechanisms that apply a single form of attention, such as spatial or channel-wise, our module employs a dual-pronged strategy. The key difference lies in the integration of both global maximum and average pooling operations, which provide a more comprehensive representation of the feature map's spatial context. This dual approach allows for a richer and more nuanced channel attention that is sensitive to both the prominent and the average spatial features, enhancing the module's ability to capture and emphasize salient features across the channel dimension.Fig. 4 Channel attention module.

Fig. 4

To obtain channel attention weights for the input feature map, the map is first compressed in the spatial dimension using global maximum pooling and global average pooling, resulting in maximum pooling feature and average pooling feature. A shared network comprising a Multi-Layer Perceptron (MLP) and a sigmoid activation function is then used to compute the channel attention weights, as shown in equation (3):(3) Wc(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))=σ(M1(M0(Favgc))+M1(M0(Fmaxc)))

Where, M0∈RC/r×C, M1∈RC×C/r, and σ represents the sigmoid activation function. Unlike channel attention, spatial attention mainly focuses on location information, and its module is shown in Fig. 5. The spatial attention module in Fig. 5 diverges from standard attention mechanisms by incorporating a unique approach that addresses specific challenges identified in the research. Unlike general attention mechanisms that may apply a uniform attention map across the entire feature space, the spatial attention module in this study is designed to focus on the location information, providing a more refined emphasis on the spatial dimensions of the feature maps.Fig. 5 Spatial attention module.

Fig. 5

The original model suffered from mixed feature information across channels and spaces, which resulted in an inability to focus on the key regions and channels of the face. To address this issue, this study introduced a convolutional attention module after the residual block to highlight the meaningful features in both channel and spatial dimensions. The channel and spatial attention modules were sequentially applied after the convolution operation, allowing the model to pay attention to both channel feature information and spatial location information. This attention mechanism emphasized the meaningful features while suppressing the irrelevant ones, enabling the model to focus on critical facial features such as the eyes, mouth, and other essential regions, as well as the key channels within the region. This resulted in the generation of more realistic face images.

3.2.2 Adaptive residual block

The adaptive residual block is a type of residual block where adaptive instance normalization is embedded between the convolution operations. The adaptive layer instance normalization combines instance normalization and layer normalization, and the specific structure is shown in Fig. 6. This technique is used to maintain the style and texture information of the image within each feature map. By preserving these details, the model can generate more realistic and visually pleasing images. Instance normalization can normalize the style of each image, but only if there is no correlation between the image channels, because it only normalizes the image feature map itself. Layer normalization, on the other hand, does not assume correlation between channels, and it does a global normalization without preserving the image style structure well.Fig. 6 Adaptive residual block.

Fig. 6

Adaptive layer instance normalization is a normalization approach that combines the strengths of instance normalization and layer normalization. By incorporating both techniques, the model is better able to control the amount of style and texture variation in subsequent residual blocks. This helps the attention-guided model to be more flexible in its approach to style and texture, as it considers both the style structure of the image and the correlation between the channels. In essence, adaptive layer instance normalization provides a way for the model to better understand the overall structure and characteristics of an image, which can lead to more accurate and realistic image generation. The equations for instance normalization and layer normalization are shown in equations (4), (5), (6), (7), (8), (9), (10), (11):(4) μX(g)=1B×M∑b=1B∑m=1Mgchw

(5) σX2(g)=1B×M∑b=1B∑m=1M(gchw−μX(g))2

(6) gˆX=g−μXσX2+ε

(7) μL(g)=1B∑b=1Bgb

(8) σL2(g)=1B×M∑b=1B(gb−μL(g))2

(9) gˆL=g−μLσL2+ε

(10) j=γ[αgˆX+(1−α)gˆL]+β

(11) α←clipu[0,1](α−δ△α)

Where RB×M×C denotes the input space, g∈RB×M×C. B and M are the spatial location, and C is the channel index. γ and β are obtained from the multilayer perceptron, α is the learning parameter, δ is the learning rate, and △α denotes the parameter update vector. α is restricted between [0, 1] in the parameter update to indicate the importance of the style for the current task, and the larger α is, the more important it is.

3.3 Discriminator structure

In discriminative networks, a 70 × 70 PatchGAN network structure is used to discriminate whether a local patch with a perceptual field of 70 × 70 is true or not. The discriminator of traditional generative adversarial networks maps the input to a real number to represent the probability of the generated image being true. In contrast, PatchGAN maps the input feature map into a 30 × 30 size output, which corresponds to the probability that the 900 70 × 70 local patches corresponding to the input feature map are true. The PatchGAN discriminator is a type of discriminator that operates at the patch level, as opposed to the whole-image level of traditional discriminators. Specifically, the discriminator convolves a patch of the input image to obtain a 30 x 30 size output, and then averages these values to obtain the final output of the discriminator. By operating at the patch level, the PatchGAN discriminator can pay more attention to image details and capture local texture information. This allows it to better distinguish between real and fake images and generate higher quality images.

3.4 Loss function

CycleGAN is a non-pairwise image translation network that uses two pairs of generators and discriminators to translate images from one class to another. Adversarial loss and cyclic consistency loss are used in both generators. Suppose I and J are two existing image domains. The generator A learns to map I to J. The discriminator DJ discriminates whether an image is real or not, and its adversarial loss function is shown in equations (12), (13).(12) maxLAGT(DJ)=Ej∼Udata(j)[logDJ(j)]+Ei∼Udata(i)[log(1−DJ(A(i)))]

(13) minLAGT(A)=Ej∼Udata(j)[logDJ(j)]+Ei∼Udata(i)[log(1−DJ(A(i)))]

This research improves the discriminative power of the discriminator by using prior knowledge that half of the input data is true and half is false. The logarithmic operation is replaced with the residual square operation using the least squares algorithm to optimize false samples and penalize false samples far from the decision boundary more strictly, resulting in better generated images. DJ(j) is changed to DJ(j)−DJ(A(i), and DJ(A(i)) is changed to DJ(A(i))−DJ(j) in the loss function, making the true and false values relative rather than absolute. The improved discriminator loss functions are presented in equations (14), (15).(14) minLRLSAGT(DJ)=12Ej∼Udata(j)∼∼Udata(i)[(DJ(j)−DJ(A(i))−1)2+(DJ(A(i))−DJ(j))2]

(15) minLRLSAGT(A)=12Ej∼Udata(j)i∼Udata(i)[(DJ(j)−DJ(A(i)))2+(DJ(A(i))−DJ(j)−1)2]

Similarly, another generator F learns the mapping from J to I. For the image generated by F, the discriminator DJ discriminates whether it is a real image or not, and then feeds DJ determination of the image generated by F to F to guide the network for training. In addition to the adversarial loss, the cyclic consistency loss is additionally introduced. In learning the mapping from I to J and J to I, the image domain I is passed through the generator A to generate the forged image domain, and then passed through F to generate the reconstructed image domain, so that the loss between them is calculated. Similarly, the cyclic consistency loss function is shown in equation (16):(16) LCJC(A,F)=Ej∼Udata(j)[∥A(F(j))−j∥1]+Ei∼data(i)[∥F(A(i))−i∥1]

Considering the previous analysis, the complete loss function for the model is represented in equation (17):(17) LGL(A,F,DI,DJ)=LRLSGAN(A,DJ,I,J)+LRLSGAN(F,DI,J,I)+λLCYC(A,F)

Where λ determines the weight of the cyclic consistency loss and LGL(A,F,DI,DJ) represents the overall loss function of the CycleGAN model, which consists of adversarial loss and cyclic consistency loss for both generators A and F, and discriminators DI and DJ. The final optimization objective of the model is shown in equation (18):(18) argminA,FmaxDI,DJLGL(A,F,DI,DJ)

4 Result analysis and discussion

4.1 Experimental environment

The experiment uses the CelebA face image dataset [24], which contains 202599 face images. In the CelebA face image dataset, each face image sample contains various attribute labels. This study conducts experiments to evaluate the algorithm's ability to generate faces of different age groups for both men and women. To prepare the data, the dataset is preprocessed using image annotation information and split into four experimental datasets based on gender and age group: young male, old male, young female, and old female. The images are resized to 256x256 using the "resize ()" function from the OpenCV2 library in Python.

This study utilizes pre-grouped datasets to train the model. The batch size is set to 1 in the experiments, and the learning rate for the first 100 epochs is set to 0.0002 with no linear decay. The weight parameter λ in equation (17) is set to 10. During training, the Adam optimizer is used for gradient descent optimization. The generator structure comprises 12 residual blocks, including 6 adaptive residual blocks. Specifically, we used Python 3.10 with the OpenCV2 library for image preprocessing, and PyTorch version 1.7.0 for implementing and training our model. The experimental environment is presented in Table 1.Table 1 Experimental environment.

Table 1Experimental environment	Description	
OS	Windows 11	
CPU	Intel Core i7 9750H, 2.6 GHz	
Memory	32G	
GPU	NVIDIA Titan X Pascal	
Pytorch	1.7.0	

4.2 Experimental process and evaluation index

This study compares the proposed face synthesis algorithm with several existing algorithm models [[25], [26], [27]] as control models. The experiments involve selecting some images in the CelebA dataset as the original input and some images as the target images, generating the corresponding face images using the face synthesis algorithm, and then testing and comparing the experimental results.

The output images obtained in the experiments were compared and analyzed subjectively by visual evaluation and quantitatively using pixel-based statistics. The peak signal-to-noise ratio (PSNR) was used as a commonly used method for quality evaluation, and the structural similarity (SSIM) was used as an objective evaluation criterion for image quality consistent with the characteristics of the human visual system.

The goal of this algorithmic model is to generate face detail features as realistically as possible, so the subjective visual effects of the observer need to be considered. In general, human eye perception is more sensitive to obvious changes in color brightness contrast and influenced by the observer's own region of interest. This is more from a global perspective and is more concerned with overall image changes. On the other hand, PSNR and SSIM are more from the changes of image pixel points, and the changes of pixel point values reflect the overall changes from the subdivision perspective, which can correct the subjective assessment effect of human eye vision from the numerical perspective, so the two types of analysis methods should be dialectically combined in the experimental process to make a comprehensive judgment of the experimental effect.

4.3 Analysis of experimental results

According to the described experimental steps, Fig. 7 displays the experimental results. The columns in the figure show the source and target face images of the CelebA face image data, along with face forgery images generated by various methods including [[25], [26], [27]] and the method proposed in this study. The results indicate that the proposed method outperforms the other methods in terms of generating more realistic and detailed human faces. Specifically, the method generates more natural overall image effect and more detailed texture information.Fig. 7 Experimental results.

Fig. 7

Table 2 shows the results of the evaluation metrics PSNR and SSIM computed after testing different models, including the literature [[25], [26], [27]], state-of-the-art literature [28], and the method proposed in this study. PSNR, a widely used objective measure for evaluating image quality, was calculated separately for four groups of experimental images: young male, old male, young female, and old female. According to the table, compared with the other three literatures, literature [28] shows competitive results, with PSNR values ranging from 14.02 dB in young women to 14.15 dB in older men. However, the proposed method consistently demonstrates superior performance across all four sub-experiments, with PSNR values of 14.24 dB for young males, 14.32 dB for old males, 14.29 dB for young females, and 14.28 dB for old females. These results indicate a higher peak signal-to-noise ratio, suggesting that the images generated by our model have a better signal strength relative to the noise present in the images.Table 2 PSNR (dB) experimental data.

Table 2Methods	Young male	Old male	Young female	Old female	
[25]	11.86	11.78	11.78	11.81	
[26]	12.43	12.36	12.17	12.32	
[27]	14.05	14.01	13.96	14.01	
[28]	14.11	14.15	14.02	14.12	
Ours	14.24	14.32	14.29	14.28	

Apart from using PSNR, the structural similarity of the images before and after the experiment was also measured using SSIM. Table 3 shows the SSIM values for the four groups of experimental images: young male, old male, young female, and old female. The SSIM scores of literatures [[25], [26], [27]] range from 0.39 to 0.43, which indicates that the original image details are not accurately represented. And literature [28] realizes a higher SSIM value of 0.43–0.44. However, there is still a gap compared with our research. The method proposed in this study were 0.47, 0.49, 0.46 and 0.48 in the respective sub-experiments, which were significantly better than those of the other compared methods. Model.Table 3 SSIM experimental data.

Table 3Methods	Young male	Old male	Young female	Old female	
[25]	0.41	0.43	0.42	0.40	
[26]	0.39	0.42	0.40	0.39	
[27]	0.43	0.41	0.42	0.42	
[28]	0.44	0.44	0.43	0.44	
Ours	0.47	0.49	0.46	0.48	

The results show that although the selected deep forgery detection techniques perform well on existing datasets, they still need to be improved compared to the proposed method. This observation highlights the high realism of the face images produced by the algorithm in this study. To summarize, based on the analysis of the experimental outcomes, the proposed method demonstrates enhanced visual effects, closely resembling real face images. Moreover, it exhibits superior performance compared to other models in quantitative evaluations. For instance, when considering experiments involving young males, the algorithm in this study shows an improvement of 20.06 % and 14.63 % in PSNR and SSIM values, respectively, compared to Ref. [25]. By combining the experimental data from Fig. 7 and Table 3 and it is evident that the algorithmic model in this study generates images that are more like the original ones, exhibiting reduced distortion and a stable structure. This improvement can be attributed to the following reasons.1) Enhancements in the network structure: The generator is improved by introducing the attention mechanism and adaptive residual block.

2) Improved loss function: The loss function is enhanced to address situations where the discriminator fails to distinguish between true and false images, even when the generator is well optimized. This improvement enhances the discriminative ability of the discriminator.

3) Training strategy: A novel training strategy based on age constraints is proposed to mitigate the impact of age variations on the generated images.

Overall, the approach presented in this study optimizes existing algorithms and processes, resulting in improved face generation quality and the generation of more realistic face images.

To further validate the effectiveness of the model proposed in this paper, an ablation study was designed. The experimental results are shown in Table 4.Table 4 Results of ablation experiment.

Table 4Model Variant	PSNR (dB)	SSIM	Description	
BM	13.55	0.38	Base CycleGAN	
AM	13.9	0.42	With Attention Mechanism	
ARB	14	0.44	With Adaptive Residual Blocks	
AC	14.1	0.45	With Age Constraints	
FM	14.24	0.47	Full Model (Proposed Method)	

The Base Model serves as a control, providing a baseline for comparison. The addition of the Attention Mechanism (AM) shows an improvement in both PSNR and SSIM, indicating that the attention mechanism helps the model focus on important facial features. The Adaptive Residual Blocks (ARB) further enhance the model's ability to capture details, as seen in the increased PSNR and SSIM. Age Constraints (AC) contribute to the stability and consistency of the generated images across different age groups, as reflected in the metrics. The Full Model (FM), which combines all the components, achieves the highest scores in both PSNR and SSIM, demonstrating the synergistic effect of the combined enhancements.

The results provide a clear demonstration of the individual and collective impact of the proposed enhancements on the performance of the face image generation algorithm. The results highlight the importance of each component in achieving high-fidelity face generation.

5 Conclusion

With the rapid advancements in deep learning technology, there is an increasing interest in Deepfakes and synthetic media within the realm of digital media production. However, face image generation algorithms based on GAN encounter practical issues such as domain inconsistency and insufficient facial details. To address these challenges, this research presents a novel face image generation algorithm that enhances CycleGAN. By optimizing the network structure, incorporating attention mechanisms with adaptive residual blocks, and refining the loss function, the algorithm enhances the quality and realism of the generated images. Additionally, a model training strategy based on age constraints is proposed to mitigate the impact of age variations. The algorithm is evaluated using the CelebA dataset in comparison to other algorithms. The proposed algorithm in this research produces more realistic face images with consistent improvements in PSNR, SSIM, and subjective visual effects compared to existing methods. It has the potential to enhance the efficiency and quality of digital media production, contributing to advancements in the industry. In summary, this face image generation algorithm, based on an improved CycleGAN, significantly improves image quality and realism through optimized network structure, refined loss function, and tailored training strategy, making it highly valuable for digital media production.

Data availability statement

The labeled dataset used to support the findings of this study are available from the corresponding author upon request. The clear portrait images in the article do not involve ethics and morality, and the data is sourced from the public dataset Google Images. And has obtained Google Image authorization for use, data can be obtained from the author.

Funding statement

This work is supported by the Sichuan Revolutionary Base Development Research Center "Research on the Smart Elderly Care Service Ecosystem in Dazhou City, a Revolutionary Old Area" (No. SLQ2023SB-26 ).

CRediT authorship contribution statement

Xi Cheng: Writing – review & editing, Writing – original draft, Software, Investigation.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests:Xi Cheng reports administrative support, article publishing charges, travel, and writing assistance were provided by 10.13039/501100004912 Sichuan University of Arts and Science. Xi Cheng reports a relationship with 10.13039/501100004912 Sichuan University of Arts and Science that includes: employment, non-financial support, and travel reimbursement. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
==== Refs
References

1 Checa D. Bustillo A. A review of immersive virtual reality serious games to enhance learning and training Multimed. Tool. Appl. 79 2020 5501 5527
2 Li Q. Sun Z. He R. Tan T. Deep supervised discrete hashing Adv. Neural Inf. Process. Syst. 30 2017 2479 2488
3 Akhtar Z. Deepfakes generation and detection: a short survey Journal of Imaging 9 1 2023 18 36662116
4 Bode L. Lees D. Golding D. The digital face and deepfakes on screen Convergence 27 4 2021 849 854
5 Karras T. Laine S. Aila T. A style-based generator architecture for generative adversarial networks Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2019 4401 4410 Long Beach, CA, USA
6 Zhu J.Y. Park T. Isola P. Efros A.A. Unpaired image-to-image translation using cycle-consistent adversarial networks Proceedings of the IEEE International Conference on Computer Vision 2017 2223 2232 Venice, Italy
7 Fedus W. Goodfellow I. Dai A.M. Maskgan: better text generation via filling in the_ arXiv preprint arXiv:1801.07736 2018
8 Yang H.C. Rahmanti A.R. Huang C.W. Li Y.C.J. How can research on artificial empathy Be enhanced by applying deepfakes J. Med. Internet Res. 24 3 2022 e29506
9 Zhang Z. Da F. Yu Y. Learning directly from synthetic point clouds for “in-the-wild” 3D face recognition Pattern Recogn. 123 2022 108394
10 Barnes C. Barraclough T. Deepfakes and synthetic media Emerging Technologies and International Security 2020 Routledge 206 220
11 Deng J. Pang G. Zhang Z. Pang Z. Yang H. Yang G. cGAN based facial expression recognition for human-robot interaction IEEE Access 7 2019 9848 9859
12 Andreini P. Bonechi S. Bianchini M. Mecocci A. Scarselli F. Image generation by GAN and style transfer for agar plate image segmentation Comput. Methods Progr. Biomed. 184 2020 105268
13 Park T. Zhu J.Y. Wang O. Lu J. Shechtman E. Efros A. Zhang R. Swapping autoencoder for deep image manipulation Adv. Neural Inf. Process. Syst. 33 2020 7198 7211
14 Yi R. Xia M. Liu Y.J. Lai Y.K. Rosin P.L. Line drawings for face portraits from photos using global and local structure based GANs IEEE Trans. Pattern Anal. Mach. Intell. 43 10 2020 3462 3475
15 Zhao D. Liu S. Gu D. Sun X. Wang L. Wei Y. Zhang H. Enhanced data-driven fault diagnosis for machines with small and unbalanced data based on variational auto-encoder Meas. Sci. Technol. 31 3 2019 035004
16 Zhang X. Wang X. Shi C. Yan Z. Li X. Kong B. …Mumtaz I. De-gan: domain embedded gan for high quality face image inpainting Pattern Recogn. 124 2022 108415
17 Li Q. Wang W. Xu C. Sun Z. Yang M.H. Learning disentangled representation for one-shot progressive face swapping IEEE Trans. Pattern Anal. Mach. Intell. 1 2024 1 17 37812559
18 Abbas F. Malah M. Babahenini M.C. Attentional conditional generative adversarial network for ambient occlusion approximation Intelligent Systems and Pattern Recognition: Second International Conference, ISPR 2022, Hammamet, Tunisia, 2022, Revised Selected Papers 2022, June Springer International Publishing Cham 349 361
19 Surya S. Setlur A. Biswas A. Negi S. ReStGAN: a step towards visually guided shopper experience via text-to-image synthesis Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision 2020 Snowmass Village, CO USA 1200 1208
20 Abdelmotaal H. Abdou A.A. Omar A.F. El-Sebaity D.M. Abdelazeem K. Pix2pix conditional generative adversarial networks for scheimpflug camera color-coded corneal tomography image generation Translational Vision Science & Technology 10 7 2021 21-21
21 Tov O. Alaluf Y. Nitzan Y. Patashnik O. Cohen-Or D. Designing an encoder for stylegan image manipulation ACM Trans. Graph. 40 4 2021 1 14
22 Harms J. Lei Y. Wang T. Zhang R. Zhou J. Tang X. …Yang X. Paired cycle‐GAN‐based image correction for quantitative cone‐beam computed tomography Medical physics 46 9 2019 3998 4009 31206709
23 Jing Y. Zhang S. Wang H. DapNet-HLA: adaptive dual-attention mechanism network based on deep learning to predict non-classical HLA binding sites Anal. Biochem. 666 2023 115075
24 Lingenfelter B. Davis S.R. Hand E.M. A quantitative analysis of labeling issues in the CelebA dataset Advances in Visual Computing: 17th International Symposium, ISVC 2022, San Diego, CA, USA, October 3–5, 2022, Proceedings, Part I 2022, December Springer International Publishing Cham 129 141
25 Chen D. Chen Q. Wu J. Yu X. Tong J. Face swapping: realistic image synthesis based on facial landmarks alignment Math. Probl Eng. 2019 2019
26 Kim J. Lee J. Zhang B.T. Smooth-swap: a simple enhancement for face-swapping with smoothness Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022 10779 10788 New Orleans, LA, USA
27 Peng B. Fan H. Wang W. Dong J. Lyu S. A unified framework for high fidelity face swap and expression reenactment IEEE Trans. Circ. Syst. Video Technol. 32 6 2021 3673 3684
28 Bai Y. Fan Y. Wang X. Zhang Y. Sun J. Yuan C. Shan Y. High-fidelity facial avatar reconstruction from monocular video with generative priors Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023 4541 4551
