
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39231992
69674
10.1038/s41598-024-69674-z
Article
Real-world video superresolution enhancement method based on the adaptive down-sampling model
Zhang Xu zx9886518@163.com

Wu Jinxin
https://ror.org/01285e189 grid.449836.4 0000 0004 0644 5924 Software Engineering Institute, Xiamen University of Technology, Xiamen, 361000 China
4 9 2024
4 9 2024
2024
14 206365 9 2023
7 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
With the 5G and the popularity of high-definition and ultrahigh-definition equipment, people have increasingly higher requirements for the resolution of images or videos. However, the transmission pressure on servers is also gradually increasing. Therefore, superresolution technology has attracted much attention in recent years. Simultaneously, with the further development of deep learning techniques, superresolution research is shifting from the calculation of traditional algorithms to the deep learning method, which exhibits a greatly superior final display. First, the traditional block-matching-3D (BM3D) algorithm is formed as the postprocessing module, which can avoid the uneven edge of GAN network recovery, make the picture appear more authentic, and improve the viewer’s subjective feelings. Next, the adaptive-downsampling model (ADM) is utilized to train models for specific camera styles. The high-resolution (HR) data sequence is subsequently downsampled to a low-resolution (LR) data sequence, enabling the superresolution algorithm to utilize this training set. This method can obtain better results and improve overall performance by 0.1~0.3 dB.

Subject terms

Engineering
Mathematics and computing
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Video superresolution (SR) is a well-known and complex problem in the field of computer vision and image processing. It involves reconstructing low-resolution videos into high-resolution videos. The applications of superresolution are vast, including medical image reconstruction1, remote sensing2, virtual reality panoramic videos3, monitoring systems4, live real-time videos, and high-definition television5. With the increasing popularity of fifth-generation mobile communication technology and high-definition display devices, the demand for larger and perfectly displayed images or videos on mobile devices is growing. However, this leads to server pressure and device rendering challenges. As a result, research on video superresolution has gained significant attention from scholars.

In the early development of the video superresolution task, single-image superresolution and interpolation reconstruction were often used due to limited hardware computing power and the absence of multicard training methods. However, with advancements in hardware computing power and multicard training availability in recent years, more scholars have started exploring video superresolution. The continuous progress in mobile communication technology has also emphasized the need for improved streaming media presentation. As a result, scholars have increasingly focused on enhancing algorithms and methods to improve the effectiveness of video superresolution recovery.

At the same time, due to the phenomenon of image distortion caused by the introduction of artifacts in super-resolution videos and images, some scholars also have algorithms that can be used as a post-processing module to process super-resolution images and videos and reduce this impact. For example, Ling et al. proposed a novel two-step post-processing algorithm for interpolation-based SRM6, Kim et al. proposes a Kernel ridge regression-based method for singleimage super-resolution7. These methods all reduce the impact of introducing distorted details into the video or image after super-resolution to a certain extent.

In recent years, several approaches driven by perception have been proposed to improve the visual quality of superresolution outcomes. One such approach employs perceptual loss8,9 to optimize superresolution models within feature spaces instead of pixel spaces. Another approach is integrating generative adversarial networks (GANs)10 into SR11,12 with aims to generate solutions that resemble natural images. To improve the recovery of texture details13, including semantic image priors has been suggested. Among these advancements, the SRGAN11 model is particularly noteworthy. This model incorporates residual blocks14 and utilizes perceptual loss within a GAN framework for optimization. By employing these techniques, the SRGAN significantly improves overall visual quality of reconstruction when compared to methods focused solely on the peak signal-to-noise ratio (PSNR).Figure 1 Overview of Degradation Model. At present, two kinds of degradation model have been developed. One is the traditional degradation model, which may be composed of a single degradation mode. The other is the second-order degradation model, which randomly selects the algorithm to randomly form the degradation mode, using the traditional degradation model as a basis. These degradation modes are essentially fixed, but the random operation is completed artificially.

Through extensive experiments, we discovered that employing the traditional algorithm to adjust the media sequence after the deep learning model’s superresolution process enables the resulting superresolution media sequence to better approximate real-world results. As a result, we incorporated the traditional BM3D algorithm as a postprocessing module to denoise the superresolution model. BM3D initially employs block matching and basic estimation, followed by practical Wiener filtering collaborative filtering, which effectively reduces noise and enhances the accuracy of image details. Our experimental results demonstrate that the BM3D-based postprocessing module proposed in this paper successfully eliminates noise introduced after superresolution, reducing the oil painting effect in images and ultimately improving visual perception and experience.

The existing data set is inadequate for simulating the degradation process in real-life scenarios. In this paper, the use of the adaptive downsampling model (ADM) to degrade the data set is proposed. The model generates mismatched HR and LR sequences, where LR sequences are downsampled in a similar manner to photos taken with a Nikon D5300 camera. By training the model to learn the downsampling kernel that matches the camera style, the training effectiveness of the model can be enhanced. Experimental results demonstrate that training the superresolution model with the data set processed by the adaptive downsampling model improves the model’s recovery performance, and the generated LR data better align with real-life scenes.

The above two methods are novel and more discussion can be found in the related work section. Compared with the super-resolution effect without using the method, the corresponding index is improved by 0.1–0.3 dB, and the overall visual effect is more realistic than before.

Related work

Degradation model

Blind superresolution is mainly used to address recovering high-resolution media sequences from unknown and complex low-resolution media sequences.The overview of degradation model is depicted in Fig. 1. The classic degradation15,16 model is generally used to create a low-resolution media sequence as input. The operation steps are as follows. First, the convolution operation is performed using the actual image and the blur kernel, then the downsampling operation is performed on the image according to the scale factor, and finally, the JPEG compression of the computer-transmitted image is used as input.1 LR=Down(HR)=[(HR∗k↓r+n)]JPEG

where Down denotes the degradation process. LR and HR represent high-resolution and low-resolution media sequences, respectively.

Blur. The blur degradation modeling portion is usually combined with a linear filter, mainly including mean filtering, Gaussian filtering, median filtering, and bilateral filtering. We employ the commonly used Gaussian filtering. The Gaussian filter samples from the Gaussian distribution. For a Gaussian blur kernel k with a size of 2u+1, (x,y)∈[-u,u] is sampled from a Gaussian distribution, so the formula is:2 k(x,y)=1Nexp(-12CTΣ-1C),C=[x,y]T

where Σ denotes the covariance matrix; C is the spatial coordinates; and N is the normalization constant.

Noise.Noise degradation typically uses additive Gaussian noise and Poisson noise. The probability density function of additive Gaussian noise is the same as that of the Gaussian distribution, and the standard deviation of the Gaussian distribution controls the influence of noise. If each channel of RGB has its own independent sampling noise, the noise synthesis of these channels consists of color noise.

Poisson noise follows the Poisson distribution. Poisson noise fundamentally exists because light is composed of discrete photons (particle properties of light). Poisson noise usually causes the variance of the pixel value to be close to the mean and the standard deviation to be proportional to the square root of the mean.

Downsampling. Downsampling is the basic operation for generating low-resolution images. We typically only consider upsampling and downsampling. This can be addressed by utilizing variety of algorithms, such as nearest neighbor interpolation, bilinear interpolation, bicubic interpolation and adjusting the size of the region. Different algorithms have different results.

JPEG compression. JPEG compression is a commonly used lossy compression technique for digital images that is applicable to both grayscale and color images. When JPEG compression is used on a color image, the color space of the image is often converted to YCbCr first, and YCbCr is then downsampled according to the principle of 4:2:0. Then, the JPEG compression algorithm is performed on the three channels of YCbCr: Y, Cb and Cr. In JPEG compression, the discrete cosine transform is processed according to the 8 * 8 pixel block. This size is chosen for faster computer processing speed. Moreover, every 8 * 8 processing block is completely unrelated to its adjacent processing blocks. For more details, see17.

In summary, the traditional methods of generating downsampled media sequences are formed by the random composition of the above modules. The methods cannot effectively simulate complex real-word situations, and the results generated by these methods are more likely to be composed of certain operations. Therefore, we use the deep learning method to learn the degradation model of the real situation. The deep learning method can better simulate real situations than methods utilizing simple heap order and random combination.

Super resolution

Superresolution is a classic problem in image processing. It is mainly categorized as video superresolution and image superresolution. Superresolution is employed to recover and reconstruct high-resolution media sequences from low-resolution media sequences. With the continuous development of hardware computing power in recent years, the in-depth study of deep learning, and the increasing popularity of high-definition and ultrahigh-definition devices, superresolution has attracted the attention of scholars11,12,18–24.

In SRCNN works 11,12,18–24, an end-to-end image superresolution convolutional neural network is proposed for the first time, effectively developing the methods related to convolutional neural networks. Various architectures and strategy designs improve the PSNR-oriented evaluation indicators. However, the PSNR-based evaluation method makes the image excessively smooth, resulting in smear and distortion. Therefore, GAN’s10 antagonistic training and perceptual loss are typically employed in subsequent research to improve the recovery realism.

The main video superresolution methods are the sliding window method and recursive method. Research in these methods is mainly focused on better utilizing the video context information to assist each frame in recovery and interframe alignment. Moreover, the long-term propagation of deep learning networks and the complexity of the real-world degradation process limit the performances of video superresolution methods.

Video super resolution metric

Most superresolution methods focus primarily on generating rich textures and sharpening edges. They use reference indicators such as MSE, PSNR, and SSIM26 to evaluate the similarity between superresolution real images.Figure 2 The process of using the postprocessing module.

In recent years, the effect of superresolution restoration in real scenarios has been further evaluated. Relevant scholars have designed the following indicators: blind/referenceless image spatial quality evaluator (BRISQUE)27 and natural image quality evaluator (NIQE)28. BRISQUE employs the brightness signal as its foundation, measures the naturalness of the image through local and adjacent normalization, and measures the deviation of the generated image based on the natural image model. NIQE is based on a set of ’quality perception’ features, and these features are fit into the multivariate Gaussian (MVG) model. The quality perception feature is derived from a simple but highly regularized natural scene statistic (NSS) model. The NIQE index of a given test image is expressed as the distance between the MVG model of the NSS feature extracted from the test image and the MVG model of the quality-aware feature extracted from the natural image corpus.

Therefore, when we evaluate the image quality, we will not use only PSNR and SSIM as in traditional studies, because these two indicators may cause the picture to lose its applicability for real scenarios. Therefore, alongside these two indicators, we also use the NIQE28 and Perceptual Score for comprehensive evaluation to better reflect the superresolution effect.

Block-matching 3D (BM3D) algorithm

The block-matching 3D (BM3D) algorithm, which can maintain image details and exhibits good noise reduction performance, has been widely used in image denoising applications. Here, we introduce some common algorithms related to BM3D. The details of other algorithms are provided in29–35.

BM3D includes two main steps: block matching and collaborative filtering. In the block matching step, the input image is divided into overlapping blocks, and the mean square error isometric metric is used to search for similar blocks in a predefined search window. It has exhibited strong effects in medical images and video denoising and has become the mainstream denoising algorithm.

Collaboration block-matching 3D (CBM3D) and variational block-matching 3D (VBM3D) algorithms related to BM3D also exist. These algorithms are based on the BM3D algorithm but introduce additional operations on its basis, ultimately enhancing the performance of BM3D. For example, the CBM3D algorithm integrates additional collaborative filtering to enhance performance. The VBM3D algorithm is an optimization task that equates the denoising problem with the minimization of the cost function. This variational method further improves the performance of VBM3D. Therefore, using traditional methods for processing can also improve the recovery effect of the model.Figure 3 Using the ADM to Generate the LR Data set.

Methodology

Superresolution is currently used to restore the media sequence distortion in the real scene. We use the BM3D algorithm as a postprocessing module to reduce this situation and improve the recovery effect. Second, the LR media sequence trained by the traditional superresolution model is usually generated by a random combination of special underground sampling methods, which cannot simulate the complex degradation process in real situations. To address this, we use the ADM to learn the degradation process in real situations and the model to generate an LR media sequence for improving the training effect of the superresolution model.Figure 4 The sketch of the block to be matched and the reference block on the actual image.

Postprocessing module based on the BM3D algorithm

We do not let this module directly process video and allow it to process only images. This is because using images for processing can improve the speed of IO and reduce the machine configuration requirements11. Therefore, after processing with this module, each frame must be reconverted into video and exported. The whole process structure is depicted in Fig. 2.

Nonlocal denoising methods can be classified as spatial methods. The transform method has also achieved good results in image denoising. Therefore, some scholars have combined the ideas of these two methods to form the BM3D denoising algorithm31.

The BM3D denoising algorithm involves two basic concepts: grouping and collaborative filtering. First, we introduce the grouping operation, which divides the input image into blocks. We select reference blocks, search for matching blocks in the relevant range, calculate the similarity between the matching blocks and the reference blocks, and classify them according to the similarity value. The similarity can be calculated here through the Euclidean distance. A hard threshold should be set simultaneously8. When the similarity between two blocks is greater than the hard threshold, these blocks can be added to the group8. In Fig. 4, yellow and pink denote the reference blocks, and rose red and blue denote the blocks to be matched. The matching blocks we indicate here can match the reference blocks. In simple terms, blocks that appear to be similar are grouped by calculating the distance as described later. During block matching, the expression of block distance between the ideal condition and noisy condition is:3 Distanceideal(ZxR,Zx)=||YxR-Yx||22(N1ht)2

where ||•||22 denotes that the distance L2-norm and the block YxR and Yx are respectively located at xR and x∈X in y, and the N1ht)2 denote a block of fixed size N1∗N1.

The above situation requires assuming that the image with or without noise is available and the value can be calculated from the image. However, if only noisy images are available, they can be calculated only from blocks with noise interference, such as:4 Distancenoisy(ZxR,Zx)=||ZxR-Zx||22(N1ht)2

where ZxR and Zx are denote that the noisy image block.

If the blocks are nonoverlapping, then the distance is a noncentral tendency chi-square random variable. The mean is:5 EDistancenoisy(ZxR,Zx)=Distanceideal(ZxR,Zx)+2σ2

The variance is:6 varDistancenoisy(ZxR,Zx)=8σ4(N1ht)2+8σ2Distanceideal(ZxR,Zx)(N1ht)2

To avoid the situation in which most probability density overlaps lead to incorrect grouping, a simple prefiltering operation is used to calculate the block distance. This is achieved by using a two-dimensional linear transformation to impose a hard threshold on the coefficients. The transformation formula is as follows:7 Distanceideal(ZxR,Zx)=||γ′(T2Dht(ZxR))-γ′(T2Dht(ZxR))||22(N1ht)2

Figure 5 ADM discriminator structure.The depicted discriminator structure is relatively simple. It consists of a 3 * 3 convolution kernel, InstanceNorm, a ReLu module, and four 5 * 5 modules. The convolution kernel, InstanceNorm, ReLu module and a 2 * 2 are composed to transfer the image to the LR media sequence.

where γ′ denotes the hard threshold operator with threshold λ2Dσ and T2Dht represents the two-dimensional linear transformation of the normalized results.

In the second part of the model, the grouping is improved based on the preliminary estimation, and collaborative Wiener filtering is performed. a) Block-level estimation. Each block is grouped first. BM is used to find block positions within the estimation range that are similar to the current block. Two groups of 3D arrays are constructed with these position sequences. One group is the original input, and the other group is the basic estimation. Cooperative Wiener filtering is then used to perform a 3D transform on the two groups. Wiener filtering is performed on noisy groups with respect to the basic estimation. The inverse 3D transform is applied to the filtered coefficients to generate the estimation of all blocks, and the block estimation is returned to the original position. b) Aggregation. The local estimation is obtained by the weighted average method, and the final estimation of the real image is calculated.The specific algorithm process is shown in Algorithm 1.

Adaptive down-sampling model (ADM)

Traditional superresolution is conducted on LR and HR image or video sequence pairs. These media sequences are constructed by preknown operations (such as bicubic downsampling). Reconstructing real-world media sequences in this way is difficult and results in a fuzzy effect. To address this problem, the ADM learns the actual degradation process through unsupervised learning. Because the ADM can estimate the unknown distribution of LR images through unpaired LR and HR sequences, its results are closer to the real situation than the traditional mixed random selection downsampling method, so it can promote the existing SR method to more accurately reconstruct various synthetic examples and actual examples36.

The deep learning model degrading high-resolution media sequences

Traditional superresolution is conducted on LR and HR image or video sequence pairs, and these media sequences are constructed by preknown operations (such as bicubic downsampling). Reconstructing the real-world media sequence in this way is difficult and will produce a fuzzy effect. To solve this problem, the ADM learns the actual degradation process through unsupervised learning. The ADM can enable the existing SR methods to more accurately reconstruct various synthetic and real examples. The specific operation process is depicted in Fig. 3.Algorithm 1 ADM for learning downsampler D

Figure 6 The downsampling CNN structure of the ADM.In the figure, C represents the number of output channels, and S represents the convolution step size. Here, regularization is performed using InstanceNorm because InstanceNorm subtracts the mean and standard deviation of the input in the depth direction. This can accelerate the training speed and enhance the nonlinear fitting ability of the network.

Figure 7 Nikon D5300 processed 320 * 180 resolution LR sequence. These images were taken with the Nikon D5300 camera and were only cropped to size.

Figure 8 HR sequence of DIV2K data set.For this part, we still use the high-definition data set of DIV2K to better control the single variable for effect comparison.

Figure 9 The loss function change in the training process.

Figure 10 The downsampling kernel change during training.

Therefore, we find that the real-world downsampling process is difficult to derive by mixing multiple downsampled modules. Therefore, we use the ADM model to train the real pictures taken by the device as input to obtain the downsampling kernel that is consistent with the real world, and then deduce the LR dataset paired with the HR dataset on the basis of this downsampling kernel.

The ADM mentioned above primarily uses the adversarial training framework to jointly optimize the sampler and discriminator. The structure of the discriminator and sampler is explained in Figs. 5 and 6.

Experiments

In this experiment, the RealBasicVSR model was trained with the HR dataset of REDS and the corresponding LR dataset generated by ADM, the number of training parameters iterations was 150000, and an NVIDIA V100 was used for training. Here we only retrain the second stage of the RealBasicVSR model. The first stage uses the trained model provided by the original author. The loss functions involved in the two stages of the model are shown below.

The first stage of training of the RealBasicVSR model uses a fidelity loss and an image cleaning loss pair generator.8 L1st=LPix+LClean

where, LPix denote that fidelity loss and LClean denote that image cleaning loss.

Then, the perceptual loss and adversarial loss are applied in the second stage to fine-tune the network.9 L2nd=LPix+LClean+λPerLPer+λadvLadv

where, LPer denote that the perceptual loss and Ladv denote that adversarial loss. then, λper=1 and λadv=5∗10-2,and the weights of the cleaning module are kept fixed.

In the comparison section, We trained RealBasicVSR37 and RealESRGAN23, BSRGAN38, DAN39, BasicVSR++40 and the original RealBasicVSR models by our method in the REDS41, DIV2K42 and UDM1043 data sets and other fields under three evaluation indicators such as NIQE28, PI44 and NRQM45 Make a comparison.

Comparison with the method not employing the postprocessing module

In this section, we analyze whether the use of a postprocessing module based on the BM3D algorithm and the absence of a postprocessing module impact whether the superresolution reconstruction results are closer to the real world.

Here, we use the LR part of the REDS data set as the input and the Real-BasicVSR model for superresolution reconstruction. We then copy a superresolution media sequence into the postprocessing module based on the proposed BM3D algorithm. To accelerate the inference speed of the model, we segmented the sequence into frames before superresolution reconstruction, so the generated results are also frames. Both the front and rear models use the same pretrained model and the same inference environment.

The specific experimental results are provided in the fifth chapter, which details the ablation experiment. In this section, we provide a specific comparison and corresponding data results and analysis.

Comparison with the method not employing the traditional degradation sequence training superresolution model

This section is based on existing data sets and the ADM. Images captured by the Nikon D5300 camera are used as the LR training set. The existing data set is used as the HR data set. The adaptive downsampling model is used to simulate the real downsampling process. The data set sequence is downsampled and transformed, and the training effect is compared.

The DIV2K data set is used as the HR data, and the photos taken by the Nikon D5300 camera are used as the LR data set. Due to the large resolution of the photos taken by the Nikon D5300, each photo is first converted to a 320 * 180 resolution. The number of training iterations of the ADM is set to 80. Then, the trained model with Nikon D5300 camera style downsampling is used to downsample the HR image sequence of the REDS data set. The data sets used in the training model are depicted in Figs. 7 and 8.

First, the high-definition images of the DIV2K data set are placed in the HR folder, the photos taken by the NikonD5300 and converted to a 320 * 180 resolution are placed in the LR folder, and the training script is run. The training proc ess generates a downsampling kernel and a function convergence diagram. The training process shows that loss _ gan gradually rises and tends to be stable. The rise of loss _ gan demonstrates that the noise generated fits the Nikon D5300 style. The loss function changes are shown in Fig. 9. This figure illustrates that the loss function change is large in the 10th round. The subsequent transformation gradually slows, indicating that the model has learned the downsampling degradation process with a specific style; that is, the model can convert HR images into LR images with a specific camera style.

After 80 iterations, the downsampling kernel changes, as shown in Fig. 10. Continuous training reveals some changes around the downsampled kernel; these are the degradation features that need to be learned. In the next operation, the trained downsampling kernel is used to downsample the HR data. Some changes occur in the edge of the sampling kernel, and the color depth of the sampling kernel center also gradually changes; the change is the largest in the 10th round and gradually stabilizes by the 40th round. After 80 iterations, the sampling kernel generation results under the reference model and the actual situation are fully aligned with the expectations, and the model also learns the downsampling style of a specific camera.Figure 11 Comparison of postprocessing modules after superresolution reconstruction.

Figure 12 Visual effect comparison with other models on DIV2K dataset. In the figure, GT represents the high resolution image used as reference.

After training is completed, the model is used to generate the data set, and the superfractional model is trained with different data sets. One group is trained with the REDS data set, and the other group is trained with the REDS data set generated by the ADM. The verification set uses one of the UDM10 sequences to compare the effects with the camera. All the models use video input. Finally, the comparison standard is performed by dividing the superdivided video into frames and then calculating the PSNR and SSIM values according to the superdivided frame and the actual GT frame comparison. We found that the superscore model trained using the ADM exhibited a small improvement in indicators, and the subjective perception was also enhanced. The relevant evaluation indicators are shown in Table 1.Table 1 The influence of the postprocessing module on the image evaluation index was compared.

Name	Metric	Non-ADM	ADM(ours)	
UDM10(1)	PSNR	MIN	24.9	25	
MAX	26.5	26.6	
SSIM	MIN	0.7664	0.7672	
MAX	0.8181	0.8220	
UDM10(2)	PSNR	24.5	27	
SSIM	0.72	0.84	
Camera	PSNR	37	38	
SSIM	0.929	0.939	
Comparison demonstrates that our proposed module can slightly improve the evaluation index.

Bold text indicates our method.

Comparison with other models

In this part, RealBasicVSR trained by our method and the original RealBasicVSR, RealESRGAN, BSRGAN, DAN, RealBasicVSR++ and other models are calculated respectively in REDS41, DIV2K42 and UDM1043 data set scenarios As shown in Tables 2, 3, and 4 for the three indicators of NIQE, NRQM, and PI, the results of the model trained by our method are slightly better than other models and original RealBasicVSR model in most scenarios. In the table, the green up arrow indicates that higher is better for this metric, the red down arrow indicates that lower is better for this metric, and if the metric in other scenarios is better than our method, we will mark it in dark green bold. If the metric generated by our method is worse than that of his original model, RealBasicVSR, we’ll show it in red. See Figs. 12, 13, 14 for a detailed visual effect.Table 2 Super Resolution Model Comparation on DIV2K Dateset Scene.

Metrix	RealESRGAN	BSRGAN	DAN	BasicVSR++	RealBasicVSR(Original)	Our method	
NRQM	6.94	6.65	5.83	6.79	6.89	6.9	
NIQE	2.27	3.25	4.57	4.89	2.43	2.20	
PI	2.66	3.3	4.37	4.05	2.77	2.65	
Bold text indicates our method, italic text indicates the best data in this metric, and text that is both italic and bold indicates that our method achieved the best result in that metric.

Table 3 Super Resolution Model Comparation on REDS Dateset Scene.

Metrix	RealESRGAN	BSRGAN	DAN	BasicVSR++	RealBasicVSR(Original)	Our method	
NRQM	6.99	6.81	4.64	6.22	6.90	6.98	
NIQE	2.14	2.71	4.92	3.71	2.42	2.43	
PI	2.58	2.95	5.14	3.75	2.76	2.73	
Bold text indicates our method, italic text indicates the best data in this metric.

Table 4 Super Resolution Model Comparation on UDM10 Dateset Scene.

Metrix	RealESRGAN	BSRGAN	DAN	BasicVSR++	RealBasicVSR(Original)	Our Method	
NRQM	6.48	6.15	4.69	6.14	6.26	6.26	
NIQE	3.62	3.96	5.67	5.68	3.91	4.32	
PI	3.58	3.90	5.49	4.77	3.82	4.03	
Bold text indicates our method, italic text indicates the best data in this metric.

Figure 13 Visual effect comparison with other models on REDS dataset..

Figure 14 Visual effect comparison with other models on UDM10 dataset. In the figure, GT represents the highresolution image used as reference.

Ablation study

To understand the contributions of the postprocessing module, we examine the performance gap between the baseline method and inserting the postprocessing module. Table 5 demonstrates that the postprocessing component has a considerable impact.

Postprocessing module based on BM3D.We found that after going through the postprocessing module, the overall display of the font was clearer than before. Considering other details, such as the comparison depicted in the second row of pictures, the lines are uneven after superresolution. After the post processing module, the lines flatten overall. The lines of the details processed by the postprocessing module after the superdivision of the display in Fig. 11 can avoid the picture jitter caused by the inaccurate alignment after the superdivision. As a result, the overall view is more appropriate for the actual situation. We compare the superresolution model with the superresolution model including the proposed postprocessing module. The proposed postprocessing module can better restore the reconstructed image details than the baseline method, improving the perceived image quality.

We propose that this processing method will theoretically perform better when processing LR images with noise. When considering the actual situation, we do not examine the noise situation; we only examine whether the module directly impacts the final results. As shown in Fig. 11 and Table 5, the comprehensive evaluation indicators have been slightly improved after the postprocessing module. However, in actual operation, the perception of the human eye is improved after the proposed module processing, especially in perceiving detail. Figure 11 demonstrate that the relevant details of the picture can be better restored by the proposed method.Table 5 Comparison of PSNR and SSIM values in two test conditions.

Name	PSNR	SSIM	NIQE	Perceptual Score	
Only SR Model	25.2	0.75	3.643	3.04	
SR Model + PostProcessing Module(Ours)	25.3	0.76	3.382	3.04	
The experimental data results demonstrate that the loss function change obtained by using the adaptive downsampling model is better than the original change. The recovery effect of the final trained model in UDM10 and camera shooting is better than that of the model trained by the data set without the ADM.

Bold text indicates our method.

In summary, as shown in Table 5, the three modules we proposed resulted in slight improvements in the four types of evaluation indicators. PSNR has increased by 0.1, SSIM has increased by 0.1, and NIQE has decreased by 0.26.

Conclusion

In this paper, we propose a postprocessing module and a method for improving the training effect of the superresolution model, thus enhancing the superresolution effect of the model in real scenes. We use the BM3D algorithm as the core of the postprocessing module and the ADM model to learn the degradation in real situations to better generate the LR media sequence. However, the method proposed in this study cannot be generalized to any scene, as we have only studied solutions in real scenes. In future work, we hope to improve the superresolution reconstruction effect in real scenes by improving the loss function.

Author contributions

All authors contributed equally to this work.

Data availibility

REDS dataset can be downloaded from https://seungjunnah.github.io/Datasets/reds.html and DIV2K dataset can be downloaded from https://data.vision.ee.ethz.ch/cvl/DIV2K/ and UDM dataset can be downloaded from https://github.com/psychopa4/PFNL In addition to the LR part of the ADM training data can be trained by shooting, other data can be downloaded at the above address.

Competing interest

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Peng, C., Lin, W.-A., Liao, H., Chellappa, R. & Zhou, S. K. Saint: Spatially aware interpolation network for medical slice synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7750–7759 (2020).
2. Luo Y Zhou L Wang S Wang Z Video satellite imagery super resolution via convolutional neural networks IEEE Geosci. Remote Sens. Lett. 2017 14 2398 2402 10.1109/LGRS.2017.2766204
Luo, Y., Zhou, L., Wang, S. & Wang, Z. Video satellite imagery super resolution via convolutional neural networks. IEEE Geosci. Remote Sens. Lett. 14, 2398–2402 (2017).10.1109/LGRS.2017.2766204
3. Fakour-Sevom, V., Guldogan, E. & Kämäräinen, J.-K. 360 panorama super-resolution using deep convolutional networks. In International Conference on Computer Vision Theory and Applications (VISAPP), vol. 1, 1 (2018).
4. Deshmukh AB Usha Rani N Fractional-grey wolf optimizer-based kernel weighted regression model for multi-view face video super resolution Int. J. Mach. Learn. Cybern. 2019 10 859 877 10.1007/s13042-017-0765-6
Deshmukh, A. B. & Usha Rani, N. Fractional-grey wolf optimizer-based kernel weighted regression model for multi-view face video super resolution. Int. J. Mach. Learn. Cybern. 10, 859–877 (2019).10.1007/s13042-017-0765-6
5. Patti AJ Sezan MI Tekalp AM Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time IEEE Trans. Image Process. 1997 6 1064 1076 10.1109/83.605404 18282997
Patti, A. J., Sezan, M. I. & Tekalp, A. M. Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Trans. Image Process. 6, 1064–1076 (1997).18282997 10.1109/83.605404
6. Ling F Post-processing of interpolation-based super-resolution mapping with morphological filtering and fraction refilling Int. J. Remote Sens. 2014 35 5251 5262 10.1080/2150704X.2014.938179
Ling, F. et al. Post-processing of interpolation-based super-resolution mapping with morphological filtering and fraction refilling. Int. J. Remote Sens. 35, 5251–5262 (2014).10.1080/2150704X.2014.938179
7. Ueki W Generative adversarial network-based post-processed image super-resolution technology for accelerating brain MRI: Comparison with compressed sensing Acta Radiol 2023 64 336 345 10.1177/02841851221076330 35118883
Ueki, W. et al. Generative adversarial network-based post-processed image super-resolution technology for accelerating brain MRI: Comparison with compressed sensing. Acta Radiol 64, 336–345 (2023).35118883 10.1177/02841851221076330
8. Johnson, J., Alahi, A. & Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, 694–711 (Springer, 2016).
9. Bruna, J., Sprechmann, P. & LeCun, Y. Super-resolution with deep convolutional sufficient statistics. arXiv preprint arXiv:1511.05666 (2015).
10. Goodfellow, I. et al. Generative adversarial nets. Adv. Neural Inf. Process. Syst. 27 (2014).
11. Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4681–4690 (2017).
12. Sajjadi, M. S., Scholkopf, B. & Hirsch, M. Enhancenet: Single image super-resolution through automated texture synthesis. In Proceedings of the IEEE international conference on computer vision, 4491–4500 (2017).
13. Wang, X., Yu, K., Dong, C. & Loy, C. C. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE conference on computer vision and pattern recognition, 606–615 (2018).
14. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770–778 (2016).
15. Elad M Feuer A Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images IEEE Trans. Image Process. 1997 6 1646 1658 10.1109/83.650118 18285235
Elad, M. & Feuer, A. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Process. 6, 1646–1658 (1997).18285235 10.1109/83.650118
16. Liu C Sun D On Bayesian adaptive video super resolution IEEE Trans. Pattern Anal. Mach. Intell. 2013 36 346 360 10.1109/TPAMI.2013.127
Liu, C. & Sun, D. On Bayesian adaptive video super resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36, 346–360 (2013).10.1109/TPAMI.2013.127
17. Shin, R. & Song, D. Jpeg-resistant adversarial images. In NIPS 2017 Workshop on Machine Learning and Computer Security, vol. 1, 8 (2017).
18. Caballero, J. et al. Real-time video super-resolution with spatio-temporal networks and motion compensation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4778–4787 (2017).
19. Dong, C., Loy, C. C., He, K. & Tang, X. Learning a deep convolutional network for image super-resolution. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part IV 13, 184–199 (Springer, 2014).
20. Kappeler A Yoo S Dai Q Katsaggelos AK Video super-resolution with convolutional neural networks IEEE Trans. Comput. Imaging 2016 2 109 122 10.1109/TCI.2016.2532323
Kappeler, A., Yoo, S., Dai, Q. & Katsaggelos, A. K. Video super-resolution with convolutional neural networks. IEEE Trans. Comput. Imaging 2, 109–122 (2016).10.1109/TCI.2016.2532323
21. Kim, J., Lee, J. K. & Lee, K. M. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1646–1654 (2016).
22. Liu, D. et al. Robust video super-resolution with learned temporal dynamics. In Proceedings of the IEEE International Conference on Computer Vision, 2507–2515 (2017).
23. Wang, X., Xie, L., Dong, C. & Shan, Y. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF international conference on computer vision, 1905–1914 (2021).
24. Tao, X., Gao, H., Liao, R., Wang, J. & Jia, J. Detail-revealing deep video super-resolution. In Proceedings of the IEEE international conference on computer vision, 4472–4480 (2017).
25. Dong C Loy CC He K Tang X Image super-resolution using deep convolutional networks IEEE Trans. Pattern Anal. Mach. Intell. 2015 38 295 307 10.1109/TPAMI.2015.2439281
Dong, C., Loy, C. C., He, K. & Tang, X. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2015).10.1109/TPAMI.2015.2439281
26. Wang Z Bovik AC Sheikh HR Simoncelli EP Image quality assessment: From error visibility to structural similarity IEEE Trans. Image Process. 2004 13 600 612 10.1109/TIP.2003.819861 15376593
Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004).15376593 10.1109/TIP.2003.819861
27. Mittal, A., Moorthy, A. K. & Bovik, A. C. No-reference image quality assessment in the spatial domain10.1109/tip.2012.2214050 (2012).
28. Mittal A Soundararajan R Bovik AC Making a "completely blind" image quality analyzer IEEE Signal Process. Lett. 2012 20 209 212 10.1109/LSP.2012.2227726
Mittal, A., Soundararajan, R. & Bovik, A. C. Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 20, 209–212 (2012).10.1109/LSP.2012.2227726
29. Danielyan, A., Vehvilainen, M., Foi, A., Katkovnik, V. & Egiazarian, K. Cross-color bm3d filtering of noisy raw data. In 2009 international workshop on local and non-local approximation in image processing, 125–129 (IEEE, 2009).
30. Danielyan A Katkovnik V Egiazarian K Bm3d frames and variational image deblurring IEEE Trans. Image Process. 2011 21 1715 1728 10.1109/TIP.2011.2176954 22128008
Danielyan, A., Katkovnik, V. & Egiazarian, K. Bm3d frames and variational image deblurring. IEEE Trans. Image Process. 21, 1715–1728 (2011).22128008 10.1109/TIP.2011.2176954
31. Dabov K Foi A Katkovnik V Egiazarian K Image denoising by sparse 3-d transform-domain collaborative filtering IEEE Trans. Image Process. 2007 16 2080 2095 10.1109/TIP.2007.901238 17688213
Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 16, 2080–2095 (2007).17688213 10.1109/TIP.2007.901238
32. Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Image denoising with block-matching and 3d filtering. In Image processing: algorithms and systems, neural networks, and machine learning, vol. 6064, 354–365 (SPIE, 2006).
33. Kostadin, D., Alessandro, F. & Karen, E. Video denoising by sparse 3d transform-domain collaborative filtering. In European signal processing conference 149, 2 (2007).
34. Dabov, K., Foi, A., Katkovnik, V. & Egiazarian, K. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In 2007 IEEE international conference on image processing, vol. 1, I–313 (IEEE, 2007).
35. Hou Y Zhao C Yang D Cheng Y Comments on” image denoising by sparse 3-d transform-domain collaborative filtering IEEE Trans. Image Process. 2010 20 268 270 20550998
Hou, Y., Zhao, C., Yang, D. & Cheng, Y. Comments on’’ image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process. 20, 268–270 (2010).20550998
36. Son S Kim J Lai W-S Yang M-H Lee KM Toward real-world super-resolution via adaptive downsampling models IEEE Trans. Pattern Anal. Mach. Intell. 2021 44 8657 8670 10.1109/TPAMI.2021.3106790
Son, S., Kim, J., Lai, W.-S., Yang, M.-H. & Lee, K. M. Toward real-world super-resolution via adaptive downsampling models. IEEE Trans. Pattern Anal. Mach. Intell. 44, 8657–8670 (2021).10.1109/TPAMI.2021.3106790
37. Chan, K. C., Zhou, S., Xu, X. & Loy, C. C. Investigating tradeoffs in real-world video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5962–5971 (2022).
38. Zhang, K., Liang, J., Van Gool, L. & Timofte, R. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4791–4800 (2021).
39. Huang Y Li S Wang L Tan T Unfolding the alternating optimization for blind super resolution Adv. Neural Inf. Process. Syst. 2020 33 5632 5643
Huang, Y. et al. Unfolding the alternating optimization for blind super resolution. Adv. Neural Inf. Process. Syst. 33, 5632–5643 (2020).
40. Chan, K. C., Zhou, S., Xu, X. & Loy, C. C. Basicvsr++: Improving video super-resolution with enhanced propagation and alignment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5972–5981 (2022).
41. Nah, S. et al. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2019).
42. Agustsson, E. & Timofte, R. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops (2017).
43. Yi, P., Wang, Z., Jiang, K., Jiang, J. & Ma, J. Progressive fusion video super-resolution network via exploiting non-local spatio-temporal correlations. In IEEE International Conference on Computer Vision (ICCV), 3106–3115 (2019).
44. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T. & Zelnik-Manor, L. The 2018 pirm challenge on perceptual image super-resolution. In Proceedings of the European conference on computer vision (ECCV) workshops, 0–0 (2018).
45. Ma C Yang C-Y Yang X Yang M-H Learning a no-reference quality metric for single-image super-resolution Comput. Vis. Image Underst. 2017 158 1 16 10.1016/j.cviu.2016.12.009
Ma, C., Yang, C.-Y., Yang, X. & Yang, M.-H. Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Underst. 158, 1–16 (2017).10.1016/j.cviu.2016.12.009
