
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

39256471
71876
10.1038/s41598-024-71876-4
Article
Research on denoising method based on temperature and humidity profile lidar
Zhang Bowen 12
Fan Guangqiang gqfan@aiofm.ac.cn

1
Zhang Tianshu tszhang@aiofm.ac.cn

1
1 grid.9227.e 0000000119573309 Anhui Institute of Optics and Fine Mechanics, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031 China
2 https://ror.org/04c4dkn09 grid.59053.3a 0000 0001 2167 9639 University of Science and Technology of China, Hefei, 230026 China
10 9 2024
10 9 2024
2024
14 2110216 6 2024
2 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Due to the fact that the vibration and pure rotational Raman signals collected by the temperature and humidity profile lidar were 3–4 orders of magnitude weaker than the Mie scattering signal, they were susceptible to electronic and white noise interference, which seriously affected the system signal-to-noise ratio. In this paper, an improved VMD-WT filtering method was adopted to extract effective signals and denoise. The processing outcome of several filtering algorithms was evaluated, and noisy signals were simulated to confirm the algorithm's efficacy. Based on the quantitative computation of evaluation indicators, such as signal-to-noise ratio, root mean square error, and correlation, the improved VMD-WT algorithm had more significant advantages in indicators such as signal-to-noise ratio. In order to further verify the robustness and adaptability of the proposed algorithm, experimental analysis of the filtering algorithm was conducted on the continuously collected temperature and humidity measured signals. The results demonstrated that the algorithm not only improved the detection range of lidar and suppressed high-altitude noise effectively, but also performed well in processing strong interference signals, like clouds, which led to a significant improvement in the atmospheric optical parameter inversion results. Furthermore, pseudo-color images of aerosols, temperature, and humidity changes over time and space have been used to further illustrate the algorithm's dependability and wide range of potential uses.

Keywords

Raman lidar
Signal processing
VMD
Wavelet denoising
Subject terms

Environmental sciences
Optics and photonics
the National Key R&D Program of China2022YFC3700400, 2022YFC3704000 2022YFC3700400, 2022YFC3704000 2022YFC3700400, 2022YFC3704000 Zhang Bowen Fan Guangqiang Zhang Tianshu the National Natural Science Foundation of China42005106, 41941011 42005106, 41941011 42005106, 41941011 Zhang Bowen Fan Guangqiang Zhang Tianshu issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Temperature and humidity profile lidar has been widely employed in the detection of environmental elements of meteorology such as aerosols and vertical profiles of temperature and humidity. Its features include high spatiotemporal resolution and long detection distance1. In practical uses, lidar detection technology still has several drawbacks. Raman scattering is three to four orders of magnitude less than Mie scattering because the laser beam is absorbed and scattered by atmospheric particles during atmospheric transmission, which causes the echo signal intensity to decrease with increasing distance. The echo signal will contain random noise in addition to being impacted by electrical interference, detector noise, and sky background radiation2. Thus, reducing interfering noise and extracting useful signals can enhance not only the system's detection capabilities but also the precision of aerosol, temperature, and humidity inversions. Moreover, these processes hold substantial importance for the field of atmospheric science research as a whole3–5.

Due to noise characteristics, the temperature and humidity profile lidar echo signal is a typical nonlinear and non-stationary signal with continuous amplitude and unpredictable phase in the time dimension. The processing of lidar echo signals has made extensive use of denoising techniques such as Wavelet Transform (WT), S-G filtering, Empirical Mode Decomposition (EMD), and Variational Mode Decomposition (VMD), together with associated deformation techniques, in recent years6–13. Mao14 processed the simulated and observed lidar signals using the local threshold wavelet method. The processed signals kept all of the small details from the initial signal collection, increasing the system's detection range. Wu et al.15 denoised single photon lidar by utilizing a filtering technique that combined soft thresholding and S-G, increasing the signal-to-noise ratio and lowering the false alarm rate. In order to achieve denoising aims, Huang et al.16 presented the EMD signal decomposition method in 1998. This method is capable of adaptively extracting Intrinsic Mode Functions (IMFs) with varying feature scales from finer time scales to coarser time scales. In order to process the original signal, Zhang et al.17 combined EMD and S-G filtering methods, obtaining as effective signals as feasible and somewhat enhancing the system signal-to-noise ratio. Dragomiretskiy et al.18 introduced the VMD filtering method in 2014. The signal has been broken down into a collection of Band-Limited Intrinsic Mode Functions (BLIMFs), each of which has the ability to identify the bandwidth and center frequency of a modal component. It is used to reconstruct the denoised signal after screening. However, the majority of research is limited to processing noise from analog signals or single-item data, they did not analyze signals that are gathered over extended periods of time or over large distances.

S-G filtering is noise-sensitive and necessitates choosing a filtering window size. EMD is susceptible to noise and sampling, leading to modal aliasing and endpoint effects, although it does not require the prior establishment of basis functions and calculation thresholds19,20. Compared to EMD, VMD is more accurate in signal separation and has a higher computational efficiency21. This work designed a joint denoising method based on VMD and WT to effectively reduce interference noise from the collected signals of temperature and humidity profile lidar: Detrended fluctuation analysis was used to determine the number of signal decomposition layers. Next, VMD layered the raw signals collected to obtain BLIMFs. The autocorrelation function was used to distinguish between high and low-frequency decomposition signals. Wavelet threshold denoising processing was then improved on high-frequency signals to remove high-frequency noise. The sliding filtering was applied to low-frequency signals to retain more effective signals. Finally, high and low-frequency signals were reconstructed to achieve denoising goals. Comparison of the improved VMD-WT method with S-G, EMD-PR, VMD-PR, VMD-IT, and VMD-DT for simulation signal processing. It was applied to process the measured signals of lidar, and the results showed that the improved VMD-WT filtering algorithm significantly improved the system signal-to-noise ratio, raised the effective detection distance, and could extract effective signals with high fidelity.

Methods

Principle of VMD

The VMD algorithm iteratively searches for the optimal solution of the variational model, based on classical Wiener filtering, and solves the variational problem, to determine the center frequency and bandwidth of each finite bandwidth intrinsic mode function. It also achieves effective separation of signals from low to high frequencies18. Its main algorithm includes the construction and solution of variational problems, with two constraints when solving. It requires both the minimum sum of bandwidth for each modal function and the requirement that the sum of all modal functions is equal to the original input signal. The modal function is22:1 si(t)=Ai(t)cos(Φi(t))

where the phase function Φi(t) is non monotonically decreasing, and the envelope Ai(t) is non-negative. Create a VMD constraint model by further solving the minimal sum of projected bandwidth for each modal function:2 min{si},{fi}∑i=1N∂tδ(t)+jπt×si(t)e-jfit22s.t.∑i=1Nsi=F

where fi is the frequency function of each mode, and F is the center frequency. To solve constrained optimization problems, the constrained variational problem is transformed into a non-variational problem. By utilizing the advantages of quadratic penalty terms and the Lagrange multiplier method, an augmented Lagrange function is introduced:3 L{si},{fi},β=α∑i=1N∂tδ(t)+jπt×si(t)e-jfit22+F(t)-∑i=1Nsi(t)22+<β(t),F(t)-∑i=1Nsi(t)>

where α is the quadratic penalty factor, and β (t) is the Lagrange multiplier.

The solution involves the use of alternating multiplication, iterative updating of si, fi, and βi, and the conversion of the time domain to the frequency domain by FFT transformation:4 s^in+1(f)←F^(f)-∑s^in+1(f)-∑sin(f)+β^n(f)21+2α(f-fi)2

When solved the center frequency, the minimum center frequency can be obtained as:5 fin+1←∫0∞f|s^in+1(f)|2df∫0∞|s^in+1(f)|2df

To reconstruct the denoised signal, the VMD Partial Reconstruction (VMD-PR) approach accumulates the low-frequency components of the decomposed signal while rejecting the high-frequency components. Nevertheless, there aren’t many useful signals in the high-frequency components of lidar signal processing, and some useful signals may be lost if they are directly discarded. Furthermore, additional processing is necessary due to the presence of noise in the low-frequency components21.

Improved WT algorithm

The hard threshold function in the WT filtering algorithm keeps the high-frequency region where the absolute value of the wavelet coefficients is higher than the threshold by comparing it with a predetermined threshold. This method may lead to significant effective signal loss as well as poor continuity performance of the processed wavelet coefficients. The soft threshold function may result in significant processing misalignment, which could lead to the loss of high-frequency data. Orthogonality, high vanishing distance, and symmetry are characteristics of an ideal wavelet function. Generally, some symmetry is given up to guarantee the compactness and orthogonality of wavelets. The threshold function is optimized to address the aforementioned issues:6 s(y)=sgn(y)×||y|-T/e|y|-|T|θ|y|γ-|T|γ+1γ|,|y|>T0,y|≤T

where the regulatory factors are γ and θ. The function demonstrates continuity at the threshold point, as depicted in Fig. 1, which solves the discontinuity issue with hard threshold functions. Furthermore, this function satisfies the requirement of being an odd function, meaning that it may process signals and produce the same result regardless of the signal's sign. This function can be tailored to fit various practical purposes by varying the adjustment parameters. This allows it to keep effective signal energy while avoiding the soft threshold function's constant error issue.Fig. 1 The filtering effect of different threshold methods.

VMD-IT and VMD-DT algorithms

The EMD-DT (EMD Direct Thresholding) method for directly thresholding decomposed signals is represented as19:7 s^(y)=∑i=LHy^i+∑i=H+1Ny^i+res

The hard threshold and soft threshold functions are:8 y^i=yi,|yi|>Ti0,|yi|≤Tiandy^i=sgn(yi)(|yi|-Ti),|yi|>Ti0,

where H and L are the high and low-order modes of IMFs, and Ti is the threshold of the i-th IMFs. Due to the threshold processing required for each point of data, the processed signal may be interrupted and become discontinuous.

The EMD-IT (EMD Interval Thresholding) filtering method for interval threshold denoising of decomposed signals11 defines the signal between the adjacent zero crossing points in IMFs as a modal unit zji=[zji,zj+1i], and its hard threshold and soft threshold functions are:9 y^i(zji)=yi(zji),|yi(rji)|>Ti0,|yi(rji)|≤Tiandy^i(zji)=sgn(yi(zji))(|yi(zji)-Ti|),|yi(rji)|>Ti0,|yi(rji)|≤Ti

where yirji is the unipolar value of the zero-crossing interval, and yi(zji) represents the signal between zji and zj+1i. This paper changes EMD to VMD decomposition signal, that is, the compared filtering method is changed to VMD-IT and VMD-DT. Figure 2 shows the processing results of two methods based on hard thresholding. The interval threshold contains more noise signals, while the signal filtering degree of the direct threshold is stronger.Fig. 2 Results of interval thresholding and direct thresholding. (a) Interval thresholding; (b) Direct thresholding.

Improved VMD-WT algorithm

Firstly, when using VMD to decompose the original signal, it is necessary to set the number of decomposition layers N, as the size of N will directly affect the decomposition results. Using Detrended Fluctuation Analysis (DFA) to calculate the long-term correlation scaling exponent of time series, to determine the decomposition level N, which can eliminate foreign trends of different orders in the time series and accurately calculate the statistical characteristics of the time series itself23. A model can be established for N and the scaling exponent α of the input signal as follows:10 N=argmaxN[α1:N≥T=J,N=1,2,3,...],J=1,α≤0.82,0.8<α≤1.03,1.0<α≤1.54,1.5<α

where T is the threshold, and J is the number of scaling exponent greater than the threshold.

The autocorrelation function is then utilized to differentiate between high and low-frequency BLIMFs since its main lobe width would comparatively narrow when the noise content of the BLIMFs component is large and the autocorrelation curve's main lobe exhibits a wavelet shape. Conversely, the random noise autocorrelation appears as abrupt pulse signals. A component of BLIMFs is categorized as high-frequency if the major lobe width of its autocorrelation curve is lower than the main lobe width of the original signal. This typically indicates that the component contains more random noise. It is thought that the other components have lower frequencies and less random noise because of their greater main lobe widths. This method makes it possible to recognize and handle the various noise-leveling components of BLIMFs with more accuracy. For high-frequency components, use enhanced wavelet thresholding, while for low-frequency components, use sliding average processing.

Finally, the filtered BLIMFs and residual components are reconstructed to obtain the final effective signal.

Comparative analysis of simulated signals

Analog simulation signal

The echo power of the Raman lidar equation can be expressed as1:11 P(z)=EcA2R2ηNβO(z)exp-∫0z[α0(r)+αN(r)]dr+noise(z)

where E is the laser energy, c is the speed of light, A is the telescope area, η is the optical transmittance, N is the nitrogen number density, β is the nitrogen Raman scattering cross-section, O(z) is the geometric factor, α0 is the extinction coefficient at the emission wavelength, αN is the extinction coefficient at the Raman frequency shift, and noise(z) is the superimposed random noise. The system parameters used for simulation calculation are shown in Table 1, and the simulation signal results are shown in Fig. 3. As the detection distance increases, the amplitude of the echo signal continuously decays, accompanied by more random noise.Table 1 Parameters for system simulation.

Parameters	Value	
Emitted wavelength	355 nm	
N2 wavelength	387 nm	
Pulse energy	100 mJ	
Differential scattering cross-section of N2	2.8 × 10−30 cm2/Sr	
Receiver diameter	300 mm	
Field of view	1 mrad	
Optical transmittance	0.3	
Dark counts per second	200	
Range resolution	7.5 m	

Fig. 3 Echo power of the simulated signal.

Simulated signal filtering test

The commonly used evaluation indicators for filtering effectiveness include: signal-to-noise ratio (SNR), which can be used to measure the degree of noise reduction in filtering algorithms. The larger the value, the smaller the proportion of noise in the signal; The root mean square error (RMSE) reflects the difference between the original pure signal and the processed signal and can reflect the high-frequency detail information of the signal. The smaller the value, the closer the two are; The waveform similarity NCC is the overall similarity of the signal waveform before and after denoising, and the closer it is to 1, the closer the denoised signal is to the original pure signal; The smoothness (S) represents the smoothness of the signal curve, and approaching 0 indicates a smoother curve; The correlation coefficient (R) can reflect the degree of correlation of the signal before and after denoising, and a value close to 1 indicates a greater correlation. Their calculation formulas are23:12 SNR=10log10∑s2(t)∑[s′(t)-s(t)]2

13 RMSE=1N∑[s′(t)-s(t)]2

14 NCC=∑s′(t)s(t)∑s′(t)2∑s(t)2

15 S=∑[s′(t+1)-s′(t)]2∑[s(t+1)-s(t)]2

16 R=∑(s′(t)-s¯′)(s(t)-s¯)∑(s′(t)-s¯′)2∑(s(t)-s¯)2

where s(t) is the original pure signal, and s′(t) is the denoised signal.

Utilize the methods in the second section to filter the noisy signal, and Fig. 4 displays the result. The performance evaluation indicators of different filtering algorithms are displayed in Fig. 5 and Table 2 has an exhaustive list of the calculation results. Figure 4 illustrates how the S-G filtering approach performs comparatively badly while processing noisy data. Because a considerable portion of spike noise is not efficiently eliminated from the signal, evaluation indicators like the signal-to-noise ratio perform noticeably worse. Due to endpoint effects and modal aliasing faults, the EMD-PR and VMD-PR filtering algorithms lose near-field signals, making it difficult to invert near-surface meteorological features. Additionally, there is a significant amount of interference noise within 3–4 km. The VMD-IT and VMD-DT filtering methods have comparable denoising effects. There are still a lot of burrs or absent near-field signals even though they still exhibit the original signal's shifting trend. The improved VMD-WT filtering method fixes the flaws in the soft and hard threshold functions in wavelet denoising while making full use of the benefits of VMD decomposition signals. In addition to reducing effective signal loss and smoothing the processed signal, this approach outperforms all other methods in terms of performance indicators. Table 2 shows that various filtering methods have improved the signal-to-noise ratio raised 8.29 dB, 9.46 dB, 11.09 dB, 11.63 dB, 11.45 dB, and 19.87 dB, respectively, when the input signal signal-to-noise ratio is 15 dB. The improved VMD-WT filtering algorithm is the most similar to the original echo signal and has the minimum RMSE among them. Furthermore, this method's denoised signal smoothness is less than 1, and its correlation is the highest—all of which makes it noticeably superior to other filtering methods. The improved VMD-WT filtering algorithm offers a realistic and effective means of separating effective signals and noise while maintaining the integrity of the echo signal, based on the aforementioned research. It possesses traits like flexibility and quick thinking. As a result, the proposed method offers benefits in noise filtering for processing temperature and humidity profile lidar echo signals.Fig. 4 The results of different filtering methods. (a) S-G; (b) EMD-PR; (c) VMD-PR; (d) VMD-IT; (e) VMD-DT; (f) VMD-WT.

Fig. 5 Evaluation metrics for different filtering algorithms.

Table 2 Evaluation for denoising effects of various filtering algorithms.

Denoising method	SNR/(dB)	RMSE	NCC/(%)	S	R	
S-G	23.29	1.13	99.77	2.28	0.9973	
EMD-PR	24.46	0.98	99.82	1.18	0.9979	
VMD-PR	26.09	0.81	99.87	1.42	0.9986	
VMD-IT	26.63	0.76	99.89	1.06	0.9987	
VMD-DT	26.45	0.78	99.88	1.01	0.9987	
VMD-WT	34.87	0.66	99.93	0.89	0.9992	

Comparative analysis of measured signals

The signal obtained by the temperature and humidity profile lidar in Guangzhou in 2023 was chosen as the processing object to verify the filtering effect. In order to detect minute variations in the atmosphere, a 300 mm aperture Cassegrain telescope with a 5-min time resolution was used to collect the echo signal from the laser lidar, which emitted at a wavelength of 355 nm. The temperature profile was inverted using the pure rotational Raman echo quotient value of 354 nm/353 nm, the relative humidity profile was inverted using the vibration Raman echo quotient value of 407 nm/387 nm, and the optical and physical characteristic parameters of aerosols were inverted using the scattering echo signal of 355 nm. It is evident from the analysis in the preceding section that the S-G, EMD-PR, and VMD-PR methods have glaring signal processing flaws. Therefore, the measured data is processed and analyzed in this section using the VMD-IT, VMD-DT, and improved VMD-WT filtering algorithms.

Single profile filtering experiment

Figure 6a–f show the filtering effect of single contour processing using VMD-IT, VMD-DT, and improved VMD-WT. The aerosol extinction coefficient processing result at 355 nm wavelength is shown in Fig. 6d. The data processed by the improved VMD-WT filtering methodology demonstrated higher smoothness and a considerable reduction in the presence of burrs in the data when compared to the other two analysis methods. It not only reduces noise interference in the data and improves its overall performance, but it also successfully maintains the slight variations in aerosol optical features, enhancing the data's accuracy and dependability—a critical component for any further analysis of aerosol characteristics. Figure 6b unequivocally demonstrates that, prior to the water vapor quotient value processing, there was significant interference in the signal area above 1.5 km. This led to a tailing drop in effective signals, which increased the inaccuracy of data inversion and adversely affected the study's accuracy. Although the improved VMD-WT algorithm did not completely filter out interference noise, compared to other algorithms, the fluctuation of noise contained in the filtered signal is greatly reduced, reflecting the further possibility of this algorithm. The data in Fig. 6e has been enhanced through filtering, preserving rich details of variations in water vapor while also guaranteeing the accuracy of the water vapor's distribution features, smooth trend, and transmission mechanism. More trustworthy data assistance for related research fields is provided by the filtered data, which can precisely discriminate the distribution of water vapor above 3 km. High-fidelity extraction of effective signals is a critical technology in Raman lidar for temperature detection. In contrast, if the filtering processing is insufficient, it may introduce a significant amount of noise, seriously impede the temperature inversion process, and even introduce misleading temperature distribution areas. Excessive filtering processing may result in the loss of important subtle inversion layer information, leading to misjudgment of key signals. But, as Fig. 6f shows, the temperature profile processed by the improved VMD-WT filtering method was displayed precisely and unambiguously in the inversion layers close to 1.6 km and 3 km, preserving all pertinent data in the process. In the meantime, possible "false inversion" regions at other altitudes have been successfully removed by the algorithm. These findings clearly show that the improved VMD-WT filtering algorithm performs well and offers great benefits when processing lidar signals, offering strong technical support for the identification of atmospheric data, including humidity and temperature.Fig. 6 The filtering effect of a single profile. (a) Echo signal at 355 nm; (b) Humidity quotient value; (c) Temperature quotient value; (d) Extinction coefficient of aerosol at 355 nm; (e) Relative humidity profile; (f) Temperature profile.

The filtering evaluation indicator is no longer useful as the measured signal of the system cannot be used for obtaining the original pure signal. To characterize the filtering effect, the system signal-to-noise ratio profile (SNRsys=P(z)/b(z) where b(z) is the background noise) must be used. Figure 7a–c display the results of the calculation. It is evident that the algorithm presented in this article has clear benefits and enhances the system signal-to-noise ratio of various signals. By comparing various filtering methods applied to different signals, the results of simulated signals and measured data calculations indicate that the improved VMD-WT denoising method can significantly increase the system signal-to-noise ratio while maintaining overall smoothness consistency with the original signal change trend, fully demonstrating the filtering method's exceptional adaptability and anti-interference ability. It is able to successfully suppress high-frequency noise and preserve effective data in the high-frequency signal while making a reasonable distinction between effective and noisy signals in the original lidar signal. Furthermore, by successfully extracting echo signals from background noise, this method can greatly increase the system's detection accuracy as well as the effective signals' detection range.Fig. 7 The filtering system SNR of a single profile. (a) Echo signal at 355 nm; (b) Humidity quotient value; (c) Temperature quotient value.

Continuous data filtering experiment

Strong interference, like clouds in atmospheric echo signals, can have an impact on the scaling exponent determined by DFA. To process the data, choose the measurements made in cloudless and cloudy weather scenarios. The result is displayed in Fig. 8. When there is no abrupt change in the detection signal in clear, cloudless weather, the scaling exponent curve changes gently, signifying that the signal has fully decomposed. It is recommended to select 7-decomposition layers to prevent excessive decomposition from slowing down the calculation as shown in Fig. 8a; in cloudy conditions, select 13-decomposition levels to prevent distortion of the scaling exponent due to excessive decomposition as shown in Fig. 8b.Fig. 8 Calculation results of DFA under different atmospheric conditions. (a) Cloudless; (b) Cloudy.

VMD-IT, VMD-DT, and improved VMD-WT filtering methods were applied to the continuously collected dataset in order to thoroughly assess the stability and adaptability of the filtering algorithm. The inversion results are displayed as pseudo color maps in Figs. 9, 10, and 11. Upon close comparison, it can be seen that the VMD-IT and VMD-DT filtering algorithms have considerable drawbacks when there are clouds or high aerosol concentrations in constantly acquired data. During processing, both of these methods will alter the extinction coefficient, resulting in ripple phenomena and band widening in the top and lower cloud layer regions. On the other hand, there was no distortion in the filtered data as a result of the strong backscatter signal of the cloud layer when the data processed by the improved VMD-WT filtering algorithm demonstrated notable improvements. The temperature, relative humidity, and aerosol extinction coefficient all show excellent spatiotemporal distribution continuity, and the burr phenomena in the signal is greatly diminished. The suppressing impact of signal noise is especially important in high-altitude areas. The pseudo-color map of relative humidity makes it evident that the water vapor detection height has increased during the day. The nighttime effective detection height has been raised to 4.8 km (from 19:00 on December 26 to 7:00 on December 27 in Fig. 11b), while the daytime effective detection height has been raised from 0.25 km (Figs. 9b and 10b) to 0.5 km (Fig. 11b) from 8:30 to 17:30 on December 27. Daytime temperature has a much wider effective detection range now that it is approximately 2 km (Fig. 11c) instead of 1 km (Figs. 9c and 10c) from 8:30 to 17:30 on December 27, and some faint signals that were completely obscured by noise have been entirely recovered. Furthermore, following processing using this method, the precise details of the humidity transfer process and inversion layer can also be fully retained, as can the concentration, diffusion, and transport processes of particulate matter in the atmospheric boundary layer. These results clearly show the improved VMD-WT filtering algorithm's superior performance in terms of long-term stability and adaptability, offering a more dependable and efficient lidar data processing solution.Fig. 9 The filtering effect of continuous data with VMD-IT. (a) Aerosol extinction coefficient; (b) Relative humidity; (c) Temperature.

Fig. 10 The filtering effect of continuous data with VMD-DT. (a) Aerosol extinction coefficient; (b) Relative humidity; (c) Temperature.

Fig. 11 The filtering effect of continuous data with improved VMD-WT. (a) Aerosol extinction coefficient; (b) Relative humidity; (c) Temperature.

The accuracy of the temperature and humidity profile lidar data obtained following improved VMD-WT processing was confirmed by contrasting it with data obtained from meteorological towers at the same location. The comparison data's time range matches that of the pseudo color map, and Fig. 12 displays the findings of the correlation analysis. It shows that the correlation between humidity and temperature is 0.9736 and 0.9341, respectively. This means that the lidar data processed using this filtering method maintains good consistency with the data from the meteorological tower, confirming the algorithm's accuracy and dependability.Fig. 12 Correlation analysis of temperature and relative humidity. (a) Relative humidity; (b) Temperature.

Conclusions

An improved wavelet threshold method was integrated with the VMD algorithm to further analyze the temperature and humidity profile lidar echo signals, addressing the inadequacies of the VMD algorithm. To ensure the accuracy and efficiency of the decomposition, firstly applied the DFA to identify the ideal number of layers for VMD decomposition. Next, high and low-frequency signals were separated based on the autocorrelation function's computation findings. An improved wavelet threshold processing method was used for the high-frequency components of the decomposed BLIMFs in order to eliminate noise while maintaining the effective signal. The goal of applying sliding filtering technology to low-frequency components was to smooth out the signal and lessen spurious interference. Ultimately, the final denoised signal was obtained by reconstructing the high and low-frequency BLIMFs components that have been treated. This method of improvement not only increased the signal processing accuracy but also successfully maintained important information in the original signal.

A comparison analysis was carried out with S-G, EMD, and other filtering algorithms to process simulated signals and measured data, respectively, in order to confirm the effectiveness of the improved VMD-WT filtering algorithm. The analysis's findings showed that the algorithm was quite robust and adaptive, and that it worked particularly well for processing long-term, non-stationary data with pulse characteristics. The improved VMD-WT method had notable advantages over previous filtering algorithms in that it can be easily integrated into data processing algorithms and does not require prior knowledge of noisy signals. When it came to denoising lidar echo signal processing, this method not only preserved a high degree of signal similarity but also greatly enhanced the system's signal-to-noise ratio, supplying precise original data for subsequent data inversion. As a result, the system's detection accuracy and range were significantly increased, offering atmospheric science researchers more accurate and dependable data assistance.

Acknowledgements

We appreciate Ruxia Ma and Zhizi Zhang for checking and improving the English spelling of this article.

Author contributions

Methodology, Z.B. and Z.T.; validation, Z.B. and F.G.; writing-original draft, Z.B.

Funding

This work was supported by the National Key R&D Program of China [2022YFC3700400, 2022YFC3704000], and the National Natural Science Foundation of China [42005106, 41941011].

Data availability

The datasets used and analyzed during the current study are available from the corresponding author upon reasonable request.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
==== Refs
References

1. Lange D Behrendt A Wulfmeyer V Compact operational tropospheric water vapor and temperature raman lidar with turbulence resolution Geophys. Res. Lett. 2019 46 24 14844 14853 10.1029/2019GL085774
Lange, D., Behrendt, A. & Wulfmeyer, V. Compact operational tropospheric water vapor and temperature raman lidar with turbulence resolution. Geophys. Res. Lett. 46(24), 14844–14853 (2019).10.1029/2019GL085774
2. Chang J Zhu L Li H Noise reduction in Lidar signal using correlation-based EMD combined with soft thresholding and roughness penalty Opt. Commun. 2018 407 290 295 10.1016/j.optcom.2017.09.063
Chang, J. et al. Noise reduction in Lidar signal using correlation-based EMD combined with soft thresholding and roughness penalty. Opt. Commun. 407, 290–295 (2018).10.1016/j.optcom.2017.09.063
3. Qin X Mao J Noise reduction for lidar returns using self-adaptive wavelet neural network Opt. Rev. 2017 24 3 416 427 10.1007/s10043-017-0337-8
Qin, X. & Mao, J. Noise reduction for lidar returns using self-adaptive wavelet neural network. Opt. Rev. 24(3), 416–427 (2017).10.1007/s10043-017-0337-8
4. Zhang B Liu Y Dong Z An optimal denoising method for spaceborne photon-counting LiDAR based on a multiscale quadtree Remote Sens. 2024 16 13 2475 10.3390/rs16132475
Zhang, B. et al. An optimal denoising method for spaceborne photon-counting LiDAR based on a multiscale quadtree. Remote Sens. 16(13), 2475 (2024).10.3390/rs16132475
5. Merjora A Maran P Optimized shuffle attention based Lidar signal denoising and temperature retrievals in the middle atmosphere Opt. Quant. Electron. 2024 56 1178 10.1007/s11082-024-07022-1
Merjora, A. & Maran, P. Optimized shuffle attention based Lidar signal denoising and temperature retrievals in the middle atmosphere. Opt. Quant. Electron. 56, 1178 (2024).10.1007/s11082-024-07022-1
6. Ehara N Sasase I Mori S Weak radar signal detection based on wavelet transform IEEE Int. Conf. Acoust. 1994 77 8 105 114
Ehara, N., Sasase, I. & Mori, S. Weak radar signal detection based on wavelet transform. IEEE Int. Conf. Acoust. 77(8), 105–114 (1994).
7. Li M Chen J Chen M Spectroscopic interferometer: Larger measurement range using wavelet threshold denoising and adaptive peak extraction Opt. Commun. 2024 557 15 130344 10.1016/j.optcom.2024.130344
Li, M., Chen, J. & Chen, M. Spectroscopic interferometer: Larger measurement range using wavelet threshold denoising and adaptive peak extraction. Opt. Commun. 557(15), 130344 (2024).10.1016/j.optcom.2024.130344
8. Zhou H Xie C Shen F Lidar signal processing method for atmospheric coherence length measurement based on the WD-ADMF Appl. Opt. 2024 63 12 3343 3348 10.1364/AO.518219 38856486
Zhou, H. et al. Lidar signal processing method for atmospheric coherence length measurement based on the WD-ADMF. Appl. Opt. 63(12), 3343–3348 (2024).38856486 10.1364/AO.518219
9. Song Y Zhou Y Liu P Research on an adaptive filter for the Mie lidar signal Appl. Opt. 2019 58 1 62 68 10.1364/AO.58.000062 30645513
Song, Y. et al. Research on an adaptive filter for the Mie lidar signal. Appl. Opt. 58(1), 62–68 (2019).30645513 10.1364/AO.58.000062
10. Sarvani M Raghunath K Rao S Lidar signal denoising methods-application to NARL Rayleigh lidar J. Opt. 2015 44 2 164 171 10.1007/s12596-015-0247-8
Sarvani, M., Raghunath, K. & Rao, S. Lidar signal denoising methods-application to NARL Rayleigh lidar. J. Opt. 44(2), 164–171 (2015).10.1007/s12596-015-0247-8
11. Yang G Liu Y Wang Y EMD interval thresholding denoising based on similarity measure to select relevant modes Signal Process. 2015 109 95 109 10.1016/j.sigpro.2014.10.038
Yang, G. et al. EMD interval thresholding denoising based on similarity measure to select relevant modes. Signal Process. 109, 95–109 (2015).10.1016/j.sigpro.2014.10.038
12. Gao H Kang J Zhang Z Enhancement of signal-to-noise ratio based on variational mode decomposition for phase-sensitive optical time domain reflectometry Acta Opt. Sin. 2023 43 21 2106002
Gao, H. et al. Enhancement of signal-to-noise ratio based on variational mode decomposition for phase-sensitive optical time domain reflectometry. Acta Opt. Sin. 43(21), 2106002 (2023).
13. Zhao L Mao J A novel lidar signal denoising method based on variational mode decomposition optimized using whale algorithm J. Appl. Phys. 2024 135 17 174501 10.1063/5.0195040
Zhao, L. & Mao, J. A novel lidar signal denoising method based on variational mode decomposition optimized using whale algorithm. J. Appl. Phys. 135(17), 174501 (2024).10.1063/5.0195040
14. Mao J Noise reduction for lidar returns using local threshold wavelet analysis Opt. Quant. Electron. 2012 43 59 68 10.1007/s11082-011-9503-6
Mao, J. Noise reduction for lidar returns using local threshold wavelet analysis. Opt. Quant. Electron. 43, 59–68 (2012).10.1007/s11082-011-9503-6
15. Wu C Xing W Xia L Improvement of detection performance on single photon lidar by EMD-based denoising method Optik 2019 181 760 767 10.1016/j.ijleo.2018.10.147
Wu, C. et al. Improvement of detection performance on single photon lidar by EMD-based denoising method. Optik 181, 760–767 (2019).10.1016/j.ijleo.2018.10.147
16. Huang N Shen Z Long S The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis Proc. Math. Phys. Eng. Sci. 1998 454 1971 903 995 10.1098/rspa.1998.0193
Huang, N. et al. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proc. Math. Phys. Eng. Sci. 454(1971), 903–995 (1998).10.1098/rspa.1998.0193
17. Zhang Y Ma X Hua D The Mie scattering lidar return signal denoising research based on EMD-DISPO Spectrosc and Spectral Analysis. 2011 31 11 2996
Zhang, Y. et al. The Mie scattering lidar return signal denoising research based on EMD-DISPO. Spectrosc and Spectral Analysis. 31(11), 2996 (2011).
18. Dragomiretskiy K Zosso D Variational mode decomposition IEEE Trans. Signal Process. 2014 62 3 531 544 10.1109/TSP.2013.2288675
Dragomiretskiy, K. & Zosso, D. Variational mode decomposition. IEEE Trans. Signal Process. 62(3), 531–544 (2014).10.1109/TSP.2013.2288675
19. Boudraa A Cexus J Saidi Z EMD-based signal noise reduction Proc. World Acad. Sci. Eng. Technol. 2013 1 1 33 37
Boudraa, A., Cexus, J. & Saidi, Z. EMD-based signal noise reduction. Proc. World Acad. Sci. Eng. Technol. 1(1), 33–37 (2013).
20. Kopsinis Y Stephen M Development of EMD-based denoising methods inspired by wavelet thresholding IEEE Trans. Signal Process. 2009 57 4 1351 1362 10.1109/TSP.2009.2013885
Kopsinis, Y. & Stephen, M. Development of EMD-based denoising methods inspired by wavelet thresholding. IEEE Trans. Signal Process. 57(4), 1351–1362 (2009).10.1109/TSP.2009.2013885
21. An X Yang J Denoising of hydropower unit vibration signal based on variational mode decomposition and approximate entropy Trans. Inst. Meas. Control 2016 38 3 282 292 10.1177/0142331215592064
An, X. & Yang, J. Denoising of hydropower unit vibration signal based on variational mode decomposition and approximate entropy. Trans. Inst. Meas. Control 38(3), 282–292 (2016).10.1177/0142331215592064
22. Li Z Li S A novel lidar signal-denoising algorithm based on sparrow search algorithm for optimal variational modal decomposition Remote Sens. 2022 14 19 4960 10.3390/rs14194960
Li, Z. et al. A novel lidar signal-denoising algorithm based on sparrow search algorithm for optimal variational modal decomposition. Remote Sens. 14(19), 4960 (2022).10.3390/rs14194960
23. Liu Y Yang G Li M Variational mode decomposition denoising combined the detrended fluctuation analysis Signal Process. 2016 125 349 364 10.1016/j.sigpro.2016.02.011
Liu, Y. et al. Variational mode decomposition denoising combined the detrended fluctuation analysis. Signal Process. 125, 349–364 (2016).10.1016/j.sigpro.2016.02.011
