
==== Front
Commun Biol
Commun Biol
Communications Biology
2399-3642
Nature Publishing Group UK London

39227646
6788
10.1038/s42003-024-06788-0
Article
Semantic redundancy-aware implicit neural compression for multidimensional biomedical image data
Ma Yifan 1
Yi Chengqiang 1
Zhou Yao 1
Wang Zhaofei 1
Zhao Yuxuan 1
Zhu Lanxin 1
Wang Jie 1
Gao Shimeng 1
Liu Jianchao 1
Yuan Xinyue 1
Wang Zhaoqiang 2
http://orcid.org/0000-0002-1325-7353
Liu Binbing liubinbing@hust.edu.cn

1
http://orcid.org/0000-0003-3764-817X
Fei Peng feipeng@hust.edu.cn

13
1 grid.33199.31 0000 0004 0368 7223 School of Optical and Electronic Information-Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan, 430074 China
2 grid.19006.3e 0000 0000 9632 6718 Department of Bioengineering, Henry Samueli School of Engineering and Applied Science, University of California, Los Angeles, Los Angeles, 90095 USA
3 grid.33199.31 0000 0004 0368 7223 Advanced Biomedical Imaging Facility Huazhong University of Science and Technology, Wuhan, Hubei 430074 China
3 9 2024
3 9 2024
2024
7 10819 2 2024
27 8 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
The surge in advanced imaging techniques has generated vast biomedical image data with diverse dimensions in space, time and spectrum, posing big challenges to conventional compression techniques in image storage, transmission, and sharing. Here, we propose an intelligent image compression approach with the first-proved semantic redundancy of biomedical data in the implicit neural function domain. This Semantic redundancy based Implicit Neural Compression guided with Saliency map (SINCS) can notably improve the compression efficiency for arbitrary-dimensional image data in terms of compression ratio and fidelity. Moreover, with weight transfer and residual entropy coding strategies, it shows improved compression speed while maintaining high quality. SINCS yields high quality compression with over 2000-fold compression ratio on 2D, 2D-T, 3D, 4D biomedical images of diverse targets ranging from single virus to entire human organs, and ensures reliable downstream tasks, such as object segmentation and quantitative analyses, to be conducted at high efficiency.

A semantic redundancy based implicit neural compression guided with saliency map approach can compress arbitrary-dimensional biomedical images of diverse targets ranging from single virus to entire human organs, and ensures reliable downstream tasks, such as object segmentation and quantitative analyses, to be conducted at high efficiency.

Subject terms

Imaging
Software
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Advanced imaging techniques in conjunction with efficient image processing approaches makes big impact on modern life science. Many biomedical applications currently require a vast amount of experimental data to be generated for various image-based analysis. For instance, studying the cytotoxic mechanisms of CAR-T cells through long-term live-cell imaging of cell morphological changes can produce several terabytes (TB) to tens of terabytes image data using high-resolution and high-throughput fluorescence microscopy systems1,2. For another instance, volumetric imaging of a mesoscale mouse brain at single-cell resolution to create a whole-brain neuron connectivity map will yield tens of terabytes image data3. Such a vast amount of image data places significant burdens on data storage, computation and sharing. For storage at limited space, these massive raw data have to be saved partially, with increased risk of data loss. Besides, due to the limited transmission bandwidth, researchers have to transfer and share data in an inefficient manner. Meanwhile, in contrast to centralized cloud storage and exchange technologies, it is crucial to achieve effective and data-specific storage compression directly at the user terminal. To this end, storage optimization on the data generation source4 and more importantly, on the downstream compression side should be studied.

The essence of compression lies in the removal of redundancy brought by the internal correlation among signals. Traditional compression methods based on analytical or statistical model explicitly remove spatial and temporal redundancy through transformations and coding, such as domain transformations5 and entropy coding6,7, to substantially compress the data. Besides the spatial and temporal redundancies, there are also plenty of semantical similarities—for example, visually-similar cells in a microscopic image or a video containing the dynamic changes of the same target, which are widely existed in diverse biomedical image data. Experiments reveal that within these visually similar images, there is also a form of redundancy which is different from classic temporal and spatial redundancies, and termed as semantic redundancy8,9. It’s difficult for traditional model-based methods to capture these relatively abstract semantic correlations, thus leaving ample space for further enhancing the compression efficiency. Meanwhile, classic compression methods, such as JPEG10, H.26411, H.26512 are designed for natural images/videos and thus perform poorly in compression of high dynamic range biomedical images. Moreover, frequency-domain-based compression methods may suffer from spectral truncation or blocking artifacts13, which affects the accuracy of downstream analysis tasks. Therefore, for biomedical images characterized by high dimensions (time, 3D space, spectrum), high dynamic range, and high structural similarities, conventional compression techniques often show limited fidelity insufficient for subsequent quantitative analyses.

Unlike model-based approaches, deep learning-based data compression techniques, such as Autoencoders14, VAE15, GAN16 etc. have recently emerged to interrogate the semantic correlation among signals. These approaches compress input data into a low-dimensional space representation which aims to eliminate semantic redundancy among data information by learning deep feature representations, and then reconstruct the original data using a decoder. Nevertheless, capturing essential data features, reducing dimensionality for optimal latent representation, and handling the burden of training with massive data for a single decoder remain highly challenging. Furthermore, when dealing with complex biomedical data, these deep learning-based supervised methods exhibit limited generalization capabilities and significant performance degradation due to generalization errors. The latest development of implicit neural representation (INR)17 utilizes a neural network to parameterize a continuous function based on the data dimensions, enabling advanced representations of 3D scenes18, images19, and videos20. Unlike traditional CNN-based approaches that utilize discrete, grid-like representations of image data, the grid-free feature of INR representation naturally fits with the continuity of target visual information. This facilitates the generation of continuous representations that enable seamless and arbitrary interpolation of visual data. In addition, with the universal approximation theorem of neural networks21,22, INR implemented with Multi-Layer Perceptron (MLP) can fit any complex function with a sufficient number of parameters, resulting in high-fidelity compression representation. INR currently has been demonstrated to be applicable for compressing natural image scenes23 as well as various multi-dimensional biomedical data for biomedical research and clinical diagnosis, including 2D images, 3D volumes24, and 4D data. By controlling the network parameters across different samples, stable compression rates and compression quality can be achieved. Moreover, INR exhibits stable performance for various data compression without the requirement of modifying the network structure specifically, which is can’t be achieved by other supervised learning-based approaches due to the generalization errors.

However, existing INR-based compression approaches require massive training time25–27 due to the processing of a large number of input coordinates when representing compression data by network optimization. As the dimensions increase, the network parameters of INR also grow exponentially, resulting in significantly increased computation28. Also, these INR approaches can’t effectively leverage the correlations between data, such as the inter-frame correlation in dynamic biomedical data and highly similar local features. Furthermore, INR fits the entire image region without specific optimization for particular signals. These challenges prevent the INR-based compression from surpassing the alternative techniques and also spurred the development of our new INR approach to overcome the limitations.

In this study, based on our first-proved semantic redundancy of biomedical data in implicit neural function domain, we propose the Semantic redundancy based Implicit Neural Compression guided with Saliency map (SINCS) approach, which explores the weight clustering effect in the implicit neural function domain and substantially accelerates the compression time of the algorithm through weight transfer. We also introduce saliency-guided compression mechanism to adaptively capture the specific structure information, thus facilitating high-fidelity compression of multi-modal biomedical images, and design residual-based entropy coding to further compress the optimized INR weights. Taken together, SINCS efficiently achieves high-fidelity compression with a high ratio up to 2000 folds for diverse multidimensional images. We have demonstrated SINCS’s superior compression performance on several large-scale biomedical image sets (2D, 2D video, 3D and 3D video) obtained from different imaging techniques (optical microscope, electron microscope, CT), proving its strong potentials for advancing a broad range of biomedical applications.

Results

High semantic correlation in implicit neural function domain of biomedical imaging data

The essence of compression is to eliminate the redundancy caused by the internal correlation among signals. Most of the existed compression methods only explore and eliminate temporal and spatial redundancy among data, ignoring semantic redundancy, which is common in multidimensional data, such as 2D picture, 2D video, 3D video, etc. The features in multidimensional biomedical images, such as organelles in the time-lapse video of a live cell, neurons in different regions of a large brain tissue image, are also correlated with semantic redundancy at both spatial and temporal dimensions. Here, we used zebrafish embryo heart as target to classify the biomedical image data into three modes for analyzing the sematic redundancy at different dimensions. In Mode 1, we analyzed the semantic redundancy along lateral dimensions using 2D plane image of the embryo heart. We employed the “patch sliding” to obtain the small patches from whole zebrafish embryo heart. Then we selected adjacent patches for compression using INR. By computing the Wilson Coefficient (Method section) to measure the distribution discrepancy of all the network parameters, we validated distributional correlation between the patches. In the domain of Implicit Neural Function (INF), the central tendency of the distribution is much more similar, whereas in the spatial domain, it presents a multi-modal distribution, as comparatively shown in Fig. 1a. The lower Average Wilson Coefficient (AWC) values calculated in INF domain, as compared with those in spatial domain, indicate much stronger sematic correlations found in INF domain. With discrepancy distribution and AWC metrics, we also analyzed the semantic redundancy along axial dimension (Mode 2) and temporal dimension (Mode 3) using the 3D image stack and 4D image video of the embryo heart, respectively. The results (2nd and 3rd rows) are consistent with those from lateral dimension (1st row), validating that biomedical images with arbitrary dimensions are all suited for being represented and compressed in the INF domain of INR. In addition to the correlation measurement of all network parameters, we calculated correlation of the different layers, and found that the hidden layers showed relatively high correlation (Supplementary Fig. 1).Fig. 1 Validation of semantic redundancy in implicit neural function domain correlation.

a Comparison of semantic correlations in three image modes. 2D, 3D and 4D (3D + T) images of the same zebrafish embryo heart were used as target, to evaluate the lateral (mode 1, top), axial (mode 2, middle), and temporal (mode 3, bottom) semantic correlations, respectively. In each mode, the images’ semantic correlations in the implicit neural function domain and spatial domain are compared through calculating their parameter histograms. The Average Wilson Coefficient (AWC) values are used as correlation metric with lower value indicating higher correlation. b Clusterings for structurally-similar data in implicit neural function and spatial domain. Blood vessels and cell nuclei are chosen to represent line-like and point-like signals, respectively. The Rayleigh Entropy (RE) values are calculated to quantify the clusterings, with lower value indicating more compact clustering of the images. Scale bars from top to bottom: 5 μm, 10 μm. c Comparison of image correlations in implicit neural function domain (AWC metric), and in spatial domain (SSIM metric). Multi-scale samples captured by different imaging techniques are compared to validate the universally-high correlations in implicit neural function domain. The human skeleton, human brain images are obtained from CT (Multi-Slice Spiral CT, Medium slices with 2.5 mm thickness) and MRI (T1-weighted MRI), respectively; the 3D images of zebrafish embryo heart and mouse brain neurons are obtained by light-sheet microscope (4×/0.13 NA illumination and 20×/0.5 NA detection for heart, 4×/0.28 NA illumination and 10×/0.3 NA detection for mouse brain neurons); the 2D subcellular images of cell nuclei and mitochondrial are captured by light-sheet microscope (20×/0.45 NA detection objective for cell nuclei and 60×/1.1 NA detection objective for mitochondrial), and the 4D subcellular images of dynamics microtubes are obtained from single objective light-sheet microscope (100×/1.5 NA for illumination and detection); the virus images are obtained from electron microscopy (an LEO (Zeiss, Oberkochen, Germany) with a Morada (Olympus) camera). Scale bars from left to right: 5 cm, 5 cm, 20 μm, 10 μm, 10 μm, 2 μm, 1 μm, 10 nm.

We further explored this internal correlation in INF domain on typical point-like (cell nuclei) and line-like signals (blood vessels), which belong to basic components in most of the biomedical data. Through “block partitioning” of large-scale data, we generated three thousand patches containing local features from different spatial positions or time points. Then we applied T-distributed Stochastic Neighbor Embedding (t-SNE)29 dimensionality reduction (Method section) to compare the clustering patterns of these images in both the spatial domain and the INF domain (Fig. 1b). In sharp contrast to the multi-model distributions in the spatial domain (right part), very unimodal distributions were found in the INF domain (left part). The Rayleigh Entropy (RE) was calculated to quantitatively evaluate their clustering characteristics (Method section). The significantly lower RE values indicated a much more compact clustering of massive samples in INF domain, also suggesting that different samples of the same type could be rapidly compressed using Meta-learning through weight transfer in the INF domain. Furthermore, such high semantic correlation represented by low AWC and RE metrics in INF domain also proved wide existence in arbitrary-dimensional images (2D, 2D-T, 3D, 4D) of diverse biomedical targets (virus, organelles, cells, animal tissues, Human organs) with different size (nano-, micro-, meso-, macro-scale) and from different imaging techniques (electron microscope, light microscope, CT, MRI), as shown in Fig. 1c. More comprehensive quantifications are provided in Supplementary Note 1 and Supplementary Figs. 2 and 3 to cross-validate this universal INF domain correlation inside and between biomedical images. In the following step, we designed saliency map-informed and meta-learning-enabled SINCS to fully utilize such implicit semantic correlations to realize high-fidelity, high-ratio compression with improved speed.

The principle of SINCS

Biomedical images can be considered as discrete sampling results of continuous spatiotemporal signals. The continuity representation ability of INR is suited for fitting the arbitrary-dimensional signals so that we can map the sampling result to high-dimensional functions, achieving “data-to-function” encoding. However, the large size of biomedical data poses a big challenge for INR network fitting. A simple solution is to increase the parameters of the INR, nevertheless this amplifies the computational complexity and leads to compromised compression efficiency. Based on the locality and repeatability of correlations, we first introduced an interlace group strategy which mitigated this issue through decomposing the massive high-dimensional functions (data) in INR. We used an adaptive grouping strategy to divide the original data into groups (Fig. 2a(i)). For time series data, we specifically made an interlace sampling along time series to divide the original data into N groups, for which both the global motion trend within groups and the high cross-correlation between groups can be retained due to the frame continuity. This temporal grouping strategy helps to reduce temporal redundancy and achieve a higher rate of compression. For patches that lack temporal correlation (obtained by “patch sliding”), the abovementioned grouping strategy can be modified to have ni = 1 in each group, which is also helpful to the elimination of spatial redundancy. With decomposition, we achieve highly-efficient compression by utilizing multiple simple implicit neural representations (Supplementary Fig. 4).Fig. 2 The workflow of SINCS compression and decompression.

a Compression pipeline of SINCS. For a given large-scale biomedical dataset, the pipelie contains: (i) the data is first effectively partitioned into several groups through an adaptive grouping strategy; (ii) saliency mechanism was introduced to realize adaptive compression fitting for biomedical data. This mechanism leverages saliency hot maps (serve as discrete probability distributions for coordinates query) to optimize the compression process,enabling targeted learning of crucial information in the dataset; (iii) Multi-Layer Perceptron (MLP) was constructed as a parameterized mapping function to fit each group data. After saliency-guided sampling, the selected coordinates are first encoded to vectors with high-frequency by positional encoding, and then fed into the MLP,achieving “data-function(weights)” encoding. Subsequently, a weight transfer with trianed parameters θ1 of 1st group data as the initial parameters and θ2 of 2nd group data for starting optimization was applied to promote network’s rapid convergence based on the high correlation between groups; (iv) after global fitting of the original data, a higher compression ratio can be further realized using weight-residual entropy coding strategy. Specifically, SINCS encoded residuals by subtracting the network parameters between neighboring networks and applying entropy coding to obtain the encoded initial parameters and residual ones. This step produces a bitstream at last. b Decompression pipeline of SINCS. In the decompression process, the weight-residual entropy decoding is adopted to convert the bitstream to original initial parameters θ1 and remaining residual parameters Δθi. Subsequently, by successively adding the residuals to the initial weight parameters, the original weight parameters are obtained for each network corresponding to each group data. By modeling forward inference and regrouping, the decompressed data can be reordered by the pre-defined grouping strategy. c The decompression procedure showing that the region-specific decompression and visualization can be readily achieved in SINCS with flexibility, owing to its coordinate-based representation.

Then, we designed a saliency-guided sampling to catch structural information of biomedical data. For each pre-defined data group, we proposed a training sampling strategy based on saliency mechanisms (Method section), permitting an adaptive compression that fitted the biomedical data with high-dimension structural features. A pre-trained saliency detection network, as illustrated in Fig. 2a(ii), generated saliency hot maps that serve as discrete probability distributions for coordinates query, thus incorporating the structure information into network optimization. This adaptive sampling strategy contributed to contrast improvement in decompressed results, as compared with conventional weighted loss optimization strategy with saliency map (Supplementary Fig. 5).

In the following step, the sampled coordination was converted to corresponding voxels’ value by Multi-Layer Perceptron (MLP), which was a parameterized mapping function to fit data of each group. After saliency-guided sampling, the selected coordinates were encoded to vectors with high-frequency by positional encoding (Fig. 2a(ii)), and then fed into the MLP. During the training process, the parameters of network were updated by the L2 loss computed from the network outputs and original inputs (see Supplementary Note 2, Supplementary Video 1 and Supplementary Table 1 for more training details).

After creating the high correlations between pre-partitioned groups, we designed a weight transfer fine-tuning strategy that adopts sequential fitting of each group data to accelerate network’s convergence. We compressed the data from the first group and obtained the optimized parameters (θ1’) for the initial network. Instead of introducing a new model, these optimized parameters served as the starting point for compressing the data from the second group. This meta-learning compression strategy minimizes the distance in INF domain between initial parameters (θ2) and optimized ones (θ2’) of the 2nd group data (Supplementary Fig. 6). It yielded significantly faster network convergence as compared with direct compression of individual groups. We have demonstrated significant correlation between the weights across various time points in the INF domain, making this weight transfer possible.

With the abovementioned procedure, SINCS successfully achieved rapid compression of arbitrary-dimensional biomedical images into INR weights in a manner of “data-function”. We further increased the compression ratio using weight-residual entropy coding based on the semantic correlation between each group data (Fig. 2a(iv) and Supplementary Table 2). SINCS encoded the residuals by subtracting the network parameters between neighboring networks and applying entropy coding to obtain the encoded initial parameters and residual ones for subsequent streamlining storage and transmission processes.

At image decompression stage (Fig. 2b), weight-residual entropy decoding was adopted to convert the highly-zipped bitstream to original initial parameter (θ1) and remaining residual parameters (Δθi). After successively adding the residuals to the initial weight parameters, we obtained the original weight parameters of each network for each group data. Through model forward inference and regrouping, the decompressed data can be reordered by the pre-defined grouping strategy. It is noteworthy that with the advantage of coordinate-based representation of INR, region-specific decompression and visualization can be readily achieved for flexible downstream biomedical tasks (Fig. 2c).

SINCS achieves high-fidelity and high-ratio compression of multidimensional biomedical data

High structural fidelity compression for 2D and 3D biomedical images

Bright-field microscopic imaging has been frequently used in biomedical research, generating huge amounts of image data30. However, bright-field images typically exhibit low contrast or contain less gradient information, making it more vulnerable to the loss of such gradient structural information during large data compression. Therefore, bright-filed images especially need compression algorithms that can offer both high-fidelity representation and high compression ratios for efficient data storage enabling convenient downstream image-based tasks such as cell segmentation. Here, we applied the SINCS to compress large field-of-view (FOV) 2D imaging data acquired by an inverted light microscope. The field of view of entire two-dimensional plane is 2.67 × 3.99 mm2, and we divided the plane into 48 patches using patch sliding, as illustrated in Fig. 3a. Each patch was then compressed using the SINCS algorithm guided by saliency map. Considering the large field-of-view information of bright-field imaging, here we take the learnable saliency map to better prioritize the signal information. This learnable saliency map is also applicable to a variety of other modalities of biomedical data, such as CT31, MRI32, TEM33, as shown in Supplementary Fig. 7 and Supplementary Table 3. To showcase the compression capability, we selected two representative patches for visualization. We further enlarged the regions of interest (ROIs) within these patches in Fig. 3b. Meanwhile, we compared our method with JPEG compression, conventional INR compression and autoencoder-based compression methods (CAE)14,23,24, demonstrating the superior visual fidelity achieved by our approach. In addition, we applied SINCS to 3D cell data. We partitioned it into multiple blocks using block sliding and rapidly compressed them through weight transfer, as illustrated in Fig. 3a. We visualized the 2D/3D decompression results of SINCS and those from other compression approaches, showing that our approach offers higher visual fidelity and better preserved cellular details (Fig. 3b). We also quantified the SINCS results using several well-established metrics, showing that it achieved big compression rate of 50 (2D) and 580 (3D), high Peak Signal-to-Noise Ratio (PSNR) of 34.5 (2D) and 38 (3D), and high Structural Similarity Index (SSIM) of 0.94 (2D) and 0.96 (3D). The comparative results in Fig. 3c have shown that SINCS have outperformed the alternative compression approaches in term of these metrics. Furthermore, we conducted image-based cell segmentation using the open-source software “Cellpose”34, to validate that the superior compression quality by SINCS also necessarily led to more accurate downstream cell analysis (Fig. 3d). This advantage is further consolidated by quantitatively comparing the Intersection over Union (IoU) scores of the segmented images and the cell counting accuracy (Fig. 3e, Method section). In addition, SINCS with meta-learning also enabled much faster compression as compared to the INR SIREN (INR-S) method that compresses the entire dataset (Supplementary Fig. 8).Fig. 3 Demonstration of SINCS compression capability on 2D bright field / 3D fluorescence images and performance on downstream tasks.

a Brief illustration of 2D data patch grouping and 3D data block grouping (The respective saliency maps are shown in the bottom right corner). Scale bars from top to bottom: 100 μm, 40 μm. b Comparison of bright-field 2D cell data and 3D cell nuclei data (labeled by GFP) reconstructed by different image compression methods. Since JPEG cannot compress 3D data, we convert 3D to 2D data for batch compression, where the data compression ratio is 130× for JPEG (limitation) and 580× for other methods. Scale bars from top to bottom: 5 μm, 5 μm. c Overall performance rating using PSNR and SSIM metrics, to show that SINCS surpass alternative compression approaches in terms of higher structural fidelity. In the box plots in c, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values. d Comparative segmentations of 2D cells and intersection over union (IoU) scores of 3D cell nuclei by different compression methods. The IoU scores are used as fidelity metric with higher score indicating higher visual fidelty. TP (True Positive): the correctly reconstructed structures; FP (FalsePositive): the incorrectly hallucinated structures; FN (False Negative): the missing details. Metrics from top to bottom: Counting accuracy, and IoU score comparision. Scale bars from top to bottom: 5 μm, 5 μm. e Histograms comparing the reconstruction accuracy of different compression methods with using 2D counting accuracy (top) and 3D IoU scores as metric (bottom).

High intensity fidelity SINCS compression on quantitative imaging data of neural activities in moving C. elegans

Long-term and high-speed Ca2+ imaging of neurons in moving specimens at high spatiotemporal resolution is useful to interrogate the behavior-related neural activities through tracking the Ca2+ density change indicated by fluorescence intensity variations35,36. Therefore, compression algorithm retaining the signal intensity profile is required in this case, to reduce the data size and also reflect the neural activity state of the samples accurately.

We used SINCS to compress the sequential images of moving C. elegans captured by light-field microscopy at a high imaging rate of 100 Hz. we achieved a compression ratio of 1500 folds (From 1.6GB to1.1MB). We further compared the SINCS performance with H.265 (Supplementary Table 4) and conventional INR on the decompression of Ca2+ indicator-labelled motor neurons. The comparative ROIs showed that while H.265 and INR-S lost some weak signals owing to the abrupt intensity variations, SINCS better fit these intensity changes because of the signal enhancement by saliency map (Fig. 4a). It should be also noted that considering the sparsity of neuron signals, we also adopted a conventional threshold-based saliency map to prevent the loss of weak signals (Supplementary Fig. 9). SINCS also outperformed other approaches with showing better resolved dense signals. Then we conducted trajectory tracking of 4 motor neurons (VB1, VB4, VB8, VA7) to investigate their intensity fluctuations during C. elegans movement (Fig. 4b). When using the intensity profiles extracted from the raw data as references, we validated that SINCS retained the intensity changes of the dynamic Ca2+signals well, surpassing the results from alternative H.265 and INR-S approaches. We further quantified the PSNR, signal preservation rate and intensity correlation based on all the motor neurons (Fig. 4c, Methods section). The inherent mapping function from coordinates to signal values and the adaptability of the saliency map for identifying low-intensity signals together allow SINCS to demonstrate intensity accuracy much higher than other approaches, thereby ensuring authentic representation of dynamic biomedical data and seamless continuation of subsequent tasks (Supplementary Video 2).Fig. 4 High intensity fidelity SINCS compression on sequential Ca2+ images of moving C. elegans allowing downstream quantification of neural activities.

a The motor neurons in an entire L4 C. elegans larva reconstructed by different compression approaches (The top right corner shows the C. elegans crawling trend with a time-coded trace). The magnified views of indicated regions show that H.265 (1430×) and INR-S (1500×) lose a considerable amount of weak signals, owing to the high signal dynamic range. In sharp contrast, SINCS preserves these weak signals perfectly. Meanwhile, SINCS reconstruction also shows spatial resolution higher than the other two approachs, notably contributing to the resolving of dense signals. The SSIM are used as structral fidelity metric with higher value indicating higher fidelity. Scale bar: 10 μm. b Spatio-termporal patterns of 4 motor neurons (VB1, VB4, VB8, VA7) reconstructed by different image compression methods. The neuron tracing trajectories are displayed on the left, indicating the dynamics of neuronal signals in spatial domain. The Ca2+ activity curves of corresponding neurons reconstructed by SINCS (red), INR-S (green) and H.265 (gray) approaches are shown on the right, and compared with the ground truth curve plotted by raw data (yellow). The intensity correlations are used as metrics to quantify the intensity fidelity of the reconstructions by diverse methods, with a higher correlation value indicating a higher stability in intensity fidelity. c Overall performance rating using PSNR (top), signal preservation (middle) and intensity correlation (bottom) metrics, to show that SINCS surpass the H.265 and INR-S in term of both high structural and intensity fidelities. In the box plots in c, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values.

SINCS compression on high-dimensional images of live cells

In long-term live biomedical imaging, a variety of dynamic biological processes, such as blood flow, heartbeat, and cell-cell interactions, occur in four (3D space + time) or even five (3D space + time + spectrum) dimensions and tend to generate tremendous amounts of data which intrinsically need to be compressed. Meanwhile, such types of high-dimensional data are accompanied with complex variations in both temporal and spatial domains, making high-fidelity and high-ratio compression especially challenging and necessary to ensure the downstream tasks being conducted accurately. We applied SINCS to 4D cell super-resolution data which were acquired using our lab-built light-sheet fluorescence microscope (LSFM) with a near isotropic resolution of ~100 nm. The entire 4D image dataset contains 180 consecutive volumes with totally generating 244 giga voxels (488 gigabytes) to record the 3D dynamics of mitochondrial within a single cell across 3 min. SINCS then achieved a 700-fold high quality compression that drastically reduced the size of the data into 697 megabytes while retained the complex outer membrane morphology. We visualized the decompressed data and the raw data in the same 3D volume rendering (Imaris 9.0) to visually examine the overall high structural fidelity by SINC compression (Fig. 5a). Then we selected three time points of the same small ROI and magnified them to compare the reconstructed details by SINCS, H.265 and INR-S (Fig. 5b). It’s noted that due to the limitation of directly compressing 4D data by H.265, we concatenated all temporal axis data along the axial axis to fit it into 3D format for testing H.265 compression (referring to H.265-S). In visual comparison, SINCS significantly outperformed other approaches, accurately visualizing the transient process of a single mitochondrion fission. The error maps and SSIM metric calculated with using raw image as references further validated that SINCS achieved significantly higher structural fidelity as compared to other approaches. The incomplete mitochondrial fission observed in the results of H.265-S might be from the concatenation of temporal and axial dimensions that led to non-uniform signal distribution and signal residues (Fig. 5b). Meanwhile, since INR-S lacked sufficient fitting ability to learn regions with low signal intensity, it also led to suboptimal structural fidelity (Fig. 5b).Fig. 5 SINCS compression on 4D super-resolution images of mitochondrial dynamics and 5D light-sheet images of CAR-T cell / tumor cell interaction.

a 3D volume renderings of GT (top) and 700× SINCS compression result (bottom) showing the overall high structural similarity by SINCS. Scale bar: 10 μm. b Comparison of mitochondrion fission process reconstructed by H.265-S (678×), INR-S (700×) and SINCS (700×).The red arrows indicate the mitochondrion fission site over 4 s. As compared to the GT from raw images, only SINCS results are capable of resolving the fine structural changes. The SSIM scores of the reconstructions by three approaches are calculated, with higher value indicating higher fidelity. Scale bar: 1 μm. c Comparative results of mitochondrion morphological changes in reconstructed cross section plane. The cross sention plane at different time points demonstrate the mitochondrion contraction and expansion at nanoscale. The IoU scores are used as fidelity metric with higher score indicating higher reconstruction fidelty during the morphological and cross-sectional area changes. TP (True Positive): the correctly reconstructed structures; FP (FalsePositive): the incorrectly hallucinated structures; FN (False Negative): the missing details. Scale bar: 1 μm. d Quantitative comparison of reconstruction accuracy at mitochondrial fission site with using European distance as metric (top) and cross-section plane using area as metric (bottom). e Visual comparison of GT and 2302× SINCS compression result of 5D light-sheet fluorescence microscopy data recording the interactions between CAR-T (labeled by GFP) and Nalm6 tumor cells (labeled by Dsred) in 20 min. Scale bar: 5 μm. f Comparison of synaptic area variation and SSIM values between GT and SINCS result. In the box plots in d and f, the line within each box represents the mean; the outer edges of the box are the 10th and 90th percentiles; and the whiskers extend to the minimum and maximum values.

We analyzed the dynamics of a selected mitochondrion at its cross-section plane to further validate the reconstruction fidelity in four dimensions. H.265-S exhibited significant morphological aberrations, likely because of its forced concatenation and fitting along the axial and temporal directions (Fig. 5c, left). In the meantime, INR-S could hardly discern the inner and outer membranes, preventing the subsequent quantitative analyses (Fig. 5c, middle). In contrast, only SINCS achieved smooth morphological changes which are nearly identical with the changes in raw image data. We further quantified the reconstruction accuracy at mitochondrial fission site (metric: European distance) and cross-section plane (metric: area) over time, as shown in the top and bottom of Fig. 5d, respectively. The results verified that the structural fidelity and time signal continuity by SINCS compression were both higher than other approaches. These advances came from our novel interlace grouping strategy that ensured global continuity and inter-frame continuity for accurate compression over time (Supplementary Video 3).

We went deeper with applying SINCS to the light-sheet fluorescence microscopy data recording the interactions between CAR-T and Nalm6 tumor cells, in which the subcellular changes of CAR-T immune synapses and tumor membranes in space, time and spectrum domains together formed a highly complex task for data compression. As we can see in Fig. 5e, while SINCS achieved an impressive compression ratio of 2302 folds, from 1TB to 455.5MB, it also enabled precise cellular morphology reconstruction and thereby accurately reproduced the complete Immunotherapy processes (Fig. 5e). We computed the variation of synaptic areas (Method section) during the interaction between CAR-T cells and Nalm6 T cells to evaluate the temporal compression quality over time. In addition, we calculated the SSIM values within the ROIs across different spectral channels (Fig. 5f), demonstrating that our approach consistently maintains high-fidelity compression performance across various spectral channels and permits reliable data analysis and validation in downstream tasks (Supplementary Video 4).

Conclusion and discussion

Both conventional model-based and emerging learning-based approaches show limited performance on the compression of biomedical images that have the features of high dimension, high dynamic range, and often require accurate downstream analysis. SINCS greatly improves the compression of biomedical data in term of performance and applicability by initiating the study on semantic redundancy of biomedical image data in INF domain. After verifying the semantic redundancy in INF domain, we then designed weight transfer optimization strategy and included saliency-guided mechanism adapted to the structural characteristics of multimodal images, making SINCS capable of high-fidelity compression of diverse biomedical data with high compression ratio and improved speed provided.

SINCS applies different grouping strategies for network training and coding based on data types (please refer to Supplementary Note 3, Supplementary Fig. 10 and Supplementary Tables 5 and 6 for more details). Then, it generates saliency maps according to the signal distribution of each group of data, which adjust the network training sampling strategy to guide better parameter allocation and achieve adaptive high-fidelity compression. Moreover, since SINCS is an implicit neural function mapping from spatial coordinates to signal values, it can innately incorporate saliency-guided mechanism within it, with significantly-improved fidelity in lateral, axial, and temporal dimensions. In addition, SINCS exhibits a superior noise suppression effect, indicating its high suppression of background noise while preserving a great amount of target signal (see Supplementary Fig. 11 for more experiments). Though SINCS is currently not as fast as traditional compression methods yet, its introduction of weight transfer optimization has effectively reduced the model’s training time, as compared with other INR-based compression methods. Also, this reduction in training time will become much more significant, and could be over one order of magnitude when processing increasingly bigger data. In the following decompression process, SINCS only requires simple forward propagation of neural networks, making the decompression speed nearly an-order-of-magnitude faster than H.265, as shown in Supplementary Table 7. It is also worth noting that, for the downstream visualization or quantitative analysis of large-scale biomedical images, multiple transmissions and decompressions may be necessary given the constraints of limited bandwidth. In a lot of practical applications, the high-quality, large-ratio compression as well as high-speed decompression by SINCS makes it outstanding from the alternative approaches.

We validate the capabilities of SINCS on several types of biomedical images, especially on 4D super-resolution microscopy data of live cells whereas high-resolution, high-fidelity, efficient compression are all required. When facing these challenging data with large size, dynamic structures and high resolution, SINCS approach notably outperforms traditional H.265 and INR-based method, rendering itself a powerful and versatile data compression and transmission tool for diverse biomedical applications. Moreover, to specifically address the loss issue for certain types of medical datasets, SINCS can achieve true lossless compression by further incorporating image residuals, as demonstrated in Supplementary Fig. 12. Despite our optimization via weight transfer strategy, the compression time remains a few orders of magnitude longer than traditional methods. In the future, we envision that the INR-based compression could be more versatile with obtaining image priors of diverse samples through meta learning37. We also anticipate the further reduction of compression training time by continuously optimizing the network design strategies38.

Methods

Network optimization strategy based on saliency mechanism

We introduce a saliency mechanism with learnable or hard saliency maps to guide the adaptive allocation of parameters in INR, bridging the gap between INR and data and achieving improved data compression fidelity, as shown in Supplementary Fig. 13. Considering the characteristics of the signal distribution, we have two different saliency maps to cope with signals of different distribution types. Specifically, for data with multiple ROIs, dense signal distributions, and a demand for high structural detail, we use Gradient-weight Class Activation Mapping (Grad-CAM)39 and Multi-Structure Region of Interest (MS-ROI)40 techniques to create learnable saliency maps for the corresponding data, assigning probability values to each grid coordinate point, reflecting its proportionate importance in the data. In the process of generating saliency maps, we first apply a convolutional layer to the original data, denoted as x (with a size of a×b), using a convolution filter of size n×n. The convolution operation is represented by Equation 1, where W represents the learned filter.yij=∑a=0n∑b=0nWabx(i+a)(j+b)

In practice, multiple filters within each layer are learned in parallel, resulting in a three-dimensional feature map as the output of the convolutional layer, where the depth represents the number of filters. Subsequently, by utilizing the learned weights between the predicted class c and the feature map d, we train the Class Activation Maps (CAM) model to obtain saliency maps that capture the significance of the predicted class distribution.Mcx,y=∑d∈Dwdcfdx,y

Pc=exp∑xyMcx,y∑cexp∑xyMcx,y

where wdc is the learned weight of class c for feature map d. Training for CAM minimizes the cross entropy between objects’ true probability distribution over classes (all mass given to the true class) and the predicted distribution. The probability Pc represents the likelihood of selecting the corresponding coordinate point for each training iteration. We employ a well-trained VGG16 model to generate learnable saliency maps. The VGG16 model is pre-trained on the Caltech-256 dataset41, which contains 256 classes of man-made and natural objects (e.g. common plants and animals, buildings). Due to the domain gap between natural images and biomedical images, the generated saliency maps may fail to meet expectations when applied to biomedical images. Therefore, we performed transfer learning by adding our labeled biological data samples to the pre-trained network weights. This process, which involves retraining the network to obtain new weights, is tailored to produce the desired saliency map effects. Besides, for sparse signals with faint intensity and lack of structural details, since the subsequent task analysis focuses only on their spatial location or intensity information, we use hard saliency maps based on threshold divisions to prevent the loss of weak signals. After obtaining the saliency maps, the coordinate vectors V guided by saliency maps are further mapped to a high-dimensional embedding space using position encoding, enhancing perceptual quality. Formally, the encoding function employed in our approach is as follows:γ(v)=(sin(20πv),cos(20πv),…,sin(2L−1πv),sin(2L−1πv)).

Here γ is the mapping of the original coordinate vector V from R to R2L and L is the number of frequencies used.

TSNE clustering dimensionality reduction and correlation analysis

For correlation analysis in implicit neural function domain and spatial domain, we adopt Average Wilson Coefficient (AWC) to compare the correlation between two discrete distributions. WC can be computed from the Kolmogorov Smirnov (KS) test formula as follows:WC=supxF1x−F2x

where F1(x) and F2(x) are the Empirical Cumulative Distribution Function (ECDF) of the two distributions, respectively. In this paper, the distributions were defined as the one-dimensional vectors reshaped from the compressed network weights and the original images, respectively. The network parameter values and image pixel values are all normalized to 0 to 1. The smaller value of AWC indicates the higher correlation in the parameter value distributions between the two samples.

To interrogate the statistic correlation among sample, we use t-SNE29 dimensionality reduction and clustering to get the data distribution, and we adapt Rayleigh entropy to quantify the degree of clustering, which is calculated as follows:H=−∑i=1n(Pi*log(Pi))

where Pi is the probability that the sample point belongs to a category in the clustering result. A lower Raleigh entropy value indicates that the data distributions are more concentrated in the clustering result.

Evaluation metrics

PSNR, SSIM and IoU scores were used in our work to evaluate the compression quality of the decompressed data with respect to the original data. (PSNR, SSIM and IoU scores are all based on single channel images.) Denoting f∈RNW×NH as the decompressed data, and g∈RNW×NH as the original data, PSNR and SSIM values were calculated using the following equations (Take a 2D image as an example):PSNR=10lgMAX2NWNH∑i=1NW∑j=1NHfij−gij2

SSIM=2μxμy+C1σxy+C2μx2+μy2+C1σx2+σy2+C2

Here MAX stands for the dynamic range of the original data. Μx, μy and σx, σy are the mean value and the standard deviation of the original data and decompressed data, respectively. And C1 and C2 are constants to avoid a zero denominator. It can be deduced that compression quality is better when the SSIM is closer to 1.

For segmentations, we utilized the open-source software “Cellpose”34 to divide the regions of interest. The threshold parameter was adjusted individually for dataset to enhance the continuity of the segmentation results. For the segmentation results of decompressed data, the ones having a counterpart in the segmentation results of original data are counted as TP, otherwise counted as FP. Conversely, the segmentation results in the original data without correspondence in the decompressed counterpart are regarded as FN. For quantitative evaluation, Intersection over Union (IoU) scores are used to numerically measure the differences in segmentation results before and after compression, where:IoU=f∩gf∪g

Here |f ∩ g| represents the size of the intersection between the decompressed data region and the original data region. |f∪g| represents the size of the union between the decompressed data region and the original data region. IoU takes values between 0 and 1, with values closer to 1 indicating a higher degree of overlap between the decompressed data and the original data.

Quantitative intensity correlation analysis of worms

We performed semi-automatic tracking of motion and intensity fluctuations in each neuron of the GT using the TrackMate Fiji Plugin42. Neurons in each volume were automatically detected by applying a circular ROI through a Difference of Gaussian (DoG) detector and then tracked using a Kalman filter. If the automatic tracking failed due to rapid neuronal movement, manual correction of missing detections and tracking errors was required. After tracking neurons in the GT dataset, we export the corresponding neuron’s position coordinates to Excel. Subsequently, leveraging a custom localization algorithm, we perform localization on results under different compression methods. This process entails extracting the average intensity value Fi of all pixels within the ROI surrounding the neuron’s coordinates, effectively representing the fluorescence intensity of that neuron. Finally, employing the same approach outlined above, we generate intensity change curves LT for neurons under different methods and compare them to the ground truth intensity change curves LR.The intensity correlation is calculated as Correlation=LT ⁄ LR, representing the degree of intensity correlation.

Cell contact area analysis

We designed an algorithm to quantitatively analyze the contact area between immune cells and cancer cells during their interaction. Initially, a deep learning-based segmentation network43 is employed to segment immune cells and target cells. Subsequently, based on the segmentation results, the image is divided into four regions: immune cells, cancer cells, background, and the boundary region. Finally, in the boundary region, distance transformation and watershed algorithms are used to obtain the segmented results of the contact area between immune cells and cancer cells. The segmentation results ensure a single-pixel thickness, enabling the conversion of pixel count into the contact area.

Sample preparation

Transgenic zebrafish lines Tg(gata 1a:dsRed;cmlc2:gfp) was used in our experiments. Embryonic fish were maintained at 3-4days post-fertilization in standard E3 medium, which was supplemented with extra 1-phenyl 2-thiourea (Sigma Aldrich) to inhibit melanogenesis. Then, the larvae were anesthetized with tricaine (3-aminobenzoic acid ethyl ester, Sigma Aldrich) and immobilized in 1% low-melting-point agarose inside a fluorinated ethylene propylene tube for further imaging.

For 2D cell data, cell cultures were prepared using homemade microchips. T-cell medium was used to replace the sterile water, and 500 μL of the medium was kept in the confocal dish to submerge the chip. Then, 60 μL of CAR-T cells at a density of 1 × 106/mL was taken and dropped in. We waited for 10 min to allow the cells to fall into the chamber. Subsequently, an equal amount of target cells was taken, and the above operation was repeated.

MCF-7 cell line that expresses GFP endogenously was used in 3D cell nuclei data compression experiment, MCF-7 cells were grown in Dulbecco’s modified eagle medium (DMEM), which were supplemented with 10% fetal bovine serum (FBS) and 1% penicillin-streptomycin. Once the cells had grown to 80–90% confluence, they were harvested by 0.25% Trypsin-EDTA treatment and resuspended in the complete medium to a suspension volume of 1 × 106 cells/mL.

The strain ZM9128 hpIs595[Pacr-2(s)::GcaMP6(f)::wCherry], expressing GcaMP6f in A- and B- class motor neurons, was used to detect neuronal activity in the moving worm. The C. elegans were cultured on standard nematode growth medium plates seeded with OP50 and maintained at 22 °C incubators until the L4 stage.

To label microtubules in live U2OS cells, we followed a previously described protocol44, in which the cells were coincubated with 4 μM PV-1 and 5 μM Tubulin-Atto 488 at 37 °C for 1 h, then the cells were washed three times with culture medium (warmed to 37 °C) and cultured at 37 °C for another 1 h. Finally, the medium was replaced with phenol red free McCoy’s 5 A medium and imaged via DR–SPIM.

For labeling mitochondria in fixed cells, U2OS cells were first transfected with Tomm20-EGFP (mito OM) or Cox4-EGFP (mitochondrial matrix) using Lipofectamine LTX according to the standard protocol and cultured at 37 °C with 5% CO2 for an additional 24 h. Before imaging, the cells were fixed with 2% glutaraldehyde for 20 min.

For multi-channel 4D biomedical data compression experiment, Acute B-lymphocytic leukemia cell line Nalm6 were cultured in RPMI 1640 medium (Gibco, Grand Island, NY, USA) containing 10% fetal bovine serum (FBS; Gibco, Grand Island, NY, USA). The lentivirus packaging cell line LentiX™293 T was cultured in DMEM medium (Gibco, Grand Island, NY, USA) supplemented with 10% FBS. CAR-T cells were pretreated with 50 nM dasatinib (Selleck, Shanghai, China) for 24 h. Due to the reversible effect of dasatinib, 50 nM dasatinib was also added to all subsequent staining, imaging, and other experimental solutions.To label microtubules, CAR-T cells were stained with the SiR-tubulin probe (SpiroChrome, Switzerland) at 2 μM final concentration and incubating for 1 h in a humidified 5% CO2 incubator at 37 °C. The cells were then washed twice with warm phosphate buffer saline (PBS) and resuspended with imaging solution, consisting of the phenol red-free 1640 medium (Gibco, Grand Island, NY, USA) supplemented 10% FBS, 25 mM HEPES (Gibco, Grand Island, NY, USA), 100 U/ml penicillin and streptomycin (Gibco, Grand Island, NY, USA), and 1 μM SYTOX™ Blue stain (Invitrogen, Waltham, MA, USA).

Data acquisition

For moving C. elegans data compression experiment, the calcium signals of worms were captured using a customed-built light-field microscope (LFM)45. A water immersion objective (LUMPlanFLN 40×/0.8w, Olympus) was used to collect the epifluorescence signals from samples with scientific camera sensor (Flash 4.0 V2, Hamamatsu). The captured light-field sequences were subsequently reconstructed by the trained VCD model45 to yield the 3D videos of the calcium signals in moving worms. The max intensity projection of the 3D reconstructions was then used for the compression.

For 2D and multi-channel 4D biomedical data compression experiment, the multi-channel fluorescently labeled Car-T cell images and bright-field cell images were captured using a customed single objective light sheet microscopy compatible with both fluorescence and bright-field capabilities, based on the IX83(Olympus) framework. The primary optical elements in this configuration include the following: Objective O1 (UPLSAPO 60×/1.35, silicone, Olympus), Objective O2 (UPLXAPO 40×/0.95, air, Olympus), and Objective O3 (AMS-AGY v2.0). This system attains a spatial resolution of 0.35 × 0.35 × 1μm (with axial resolution enhancement through post-processing). For bright-field cell data, illumination was provided by LED light sources, and image acquisition was performed using an Andor camera with an exposure time of 20 ms. For multi-channel fluorescently labeled Car-T cell data, excitation was conducted using lasers at 488 and 561 nm, with the 488-channel being captured by a Hamamatsu camera with a 200-ms exposure time, and the 561-channel requiring a 2000-ms exposure time. The color filter for channel 488 is MF525-39, and the color filter for channel 561 is FBH600-40.

For static 3D biomedical data experiment, the 3D cell nuclei data was collected by a customed dual-objective light sheet microscopy. The primary optical elements in this configuration include the following: illumination objective (Mitutoyo Plan Apo Infinity Corrected Long WD Objective 20×/0.42, air), and detection objective (UPLFLN20XPH 20×/0.5, Olympus). This system attains a spatial resolution of 0.325 × 0.325 × 0.5μm. The excitation source of the system is a laser with a wavelength of 405 nm, while the sCMOS camera (Orca Flash4.0 v.3, Hamamatsu) acquires data with an exposure time of 20 ms.

For 4D biomedical data experiment, 4D cell super-resolution data was collected by a customed dual-objective light sheet microscopy, followed by post-processing image enhancement using an ID neural network46. The fluorescence signals generated within the specimen were collected by a detection objective (LUMFLN 60×/1.1 W, Olympus). The resolution of the system is 97 × 97 × 450 nm. The sCMOS camera (Orca Flash4.0 v.3, Hamamatsu) exposure was precisely triggered with a minimal 2-ms delay to effectively reduce motion blur, and the camera recorded the plane images at a rate of up to 1,000 fps.

The CT, MRI, and TEM data used in our experiments were taken from publicly available datasets. The CT images from the publicly available dataset31, MRI images from the publicly available dataset32, and TEM images from the Virus Image Dataset33.

Supplementary information

Peer Review File

Supplementary materials

Description of additional supplementary materials

Supplementary Data 1

Supplementary Movie1

Supplementary Movie2

Supplementary Movie3

Supplementary Movie4

nr-reporting-summary

Supplementary information

The online version contains supplementary material available at 10.1038/s42003-024-06788-0.

Acknowledgements

We are grateful to Dr. Zhang Meng and Dr. Yuhui Zhang for providing us the fluorescent cell samples. This work was supported by the funding from National Natural Science Foundation of China (T2225014, 21927802). National Key Research and Development Program of China (2022YFC3401102).

Author contributions

P.F., Y.M. and B.L. conceived the idea. P.F., and B.L. oversaw the project. Y.Z.X., S.G., L.Z., Y.Z., J.W. and Z.W.Q. developed the optical setups and acquired the experimental images. Y.M., C.Y., X.Y., J.L. and Z.W.F. developed the programs. Y.M. and C.Y. processed the images. Y.M., C.Y., B.L. and P.F. analyzed the data and wrote the paper.

Peer review

Peer review information

Communications Biology thanks the anonymous reviewers for their contribution to the peer review of this work. Primary Handling Editors: Joao Valente. A peer review file is available.

Data availability

The datasets generated and analyzed in this study are available from the corresponding authors upon reasonable request, and source data underlying all graphs in the manuscript can be found in the Supplementary Data 1.

Code availability

Customized SINCS program implemented in current study is available at https://github.com/feilab-hust/SINCS and Zenodo (10.5281/zenodo.13328988)47.

Competing interests

The authors declare no competing interests.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Yifan Ma, Chengqiang Yi.
==== Refs
References

1. Dekkers JF Uncovering the mode of action of engineered T cells in patient cancer organoids Nat. Biotechnol. 2023 41 60 69 10.1038/s41587-022-01397-w 35879361
Dekkers, J. F. et al. Uncovering the mode of action of engineered T cells in patient cancer organoids. Nat. Biotechnol. 41, 60–69 (2023).35879361 10.1038/s41587-022-01397-w
2. Han Y Li M Qiu F Zhang M Zhang Y-H Cell-permeable organic fluorescent probes for live-cell long-term super-resolution imaging reveal lysosome-mitochondrion interactions Nat. Commun. 2017 8 1307 10.1038/s41467-017-01503-6 29101340
Han, Y., Li, M., Qiu, F., Zhang, M. & Zhang, Y.-H. Cell-permeable organic fluorescent probes for live-cell long-term super-resolution imaging reveal lysosome-mitochondrion interactions. Nat. Commun. 8, 1307 (2017).29101340 10.1038/s41467-017-01503-6
3. Li A Micro-optical sectioning tomography to obtain a high-resolution atlas of the mouse brain Science 2010 330 1404 1408 10.1126/science.1191776 21051596
Li, A. et al. Micro-optical sectioning tomography to obtain a high-resolution atlas of the mouse brain. Science 330, 1404–1408 (2010).21051596 10.1126/science.1191776
4. Guo S Smart imaging to empower brain-wide neuroscience at single-cell levels Brain Inform. 2022 9 10 10.1186/s40708-022-00158-4 35543774
Guo, S. et al. Smart imaging to empower brain-wide neuroscience at single-cell levels. Brain Inform. 9, 10 (2022).35543774 10.1186/s40708-022-00158-4
5. Khayam SA The discrete cosine transform (DCT): theory and application Mich. State Univ. 2003 114 31
Khayam, S. A. The discrete cosine transform (DCT): theory and application. Mich. State Univ. 114, 31 (2003).
6. Sze V Budagavi M High throughput CABAC entropy coding in HEVC IEEE Trans. Circuits Syst. Video Technol. 2012 22 1778 1791 10.1109/TCSVT.2012.2221526
Sze, V. & Budagavi, M. High throughput CABAC entropy coding in HEVC. IEEE Trans. Circuits Syst. Video Technol. 22, 1778–1791 (2012).10.1109/TCSVT.2012.2221526
7. Knuth DE Dynamic huffman coding J. Algorithms 1985 6 163 180 10.1016/0196-6774(85)90036-7
Knuth, D. E. Dynamic huffman coding. J. Algorithms 6, 163–180 (1985).10.1016/0196-6774(85)90036-7
8. Pesquita C Faria D Falcao AO Lord P Couto FM Semantic similarity in biomedical ontologies PLoS Comput. Biol. 2009 5 e1000443 10.1371/journal.pcbi.1000443 19649320
Pesquita, C., Faria, D., Falcao, A. O., Lord, P. & Couto, F. M. Semantic similarity in biomedical ontologies. PLoS Comput. Biol. 5, e1000443 (2009).19649320 10.1371/journal.pcbi.1000443
9. Chandrasekaran D Mago V Evolution of semantic similarity—a survey ACM Comput. Surv. (CSUR) 2021 54 1 37 10.1145/3440755
Chandrasekaran, D. & Mago, V. Evolution of semantic similarity—a survey. ACM Comput. Surv. (CSUR) 54, 1–37 (2021).10.1145/3440755
10. Wallace, G.K.J.I.t.o.c.e. The JPEG still picture compression standard. 38, xviii-xxxiv (1992).
11. Wiegand T Sullivan GJ Bjontegaard G Luthra A Overview of the H. 264/AVC video coding standard IEEE Trans. circuits Syst. video Technol. 2003 13 560 576 10.1109/TCSVT.2003.815165
Wiegand, T., Sullivan, G. J., Bjontegaard, G. & Luthra, A. Overview of the H. 264/AVC video coding standard. IEEE Trans. circuits Syst. video Technol. 13, 560–576 (2003).10.1109/TCSVT.2003.815165
12. Sullivan GJ Ohm J-R Han W-J Wiegand T Overview of the high efficiency video coding (HEVC) standard IEEE Trans. circuits Syst. video Technol. 2012 22 1649 1668 10.1109/TCSVT.2012.2221191
Sullivan, G. J., Ohm, J.-R., Han, W.-J. & Wiegand, T. Overview of the high efficiency video coding (HEVC) standard. IEEE Trans. circuits Syst. video Technol. 22, 1649–1668 (2012).10.1109/TCSVT.2012.2221191
13. Dong, C., Deng, Y., Loy, C. C. & Tang, X. in Proceedings of the IEEE international conference on computer vision 576-584 (2015).
14. Theis, L., Shi, W., Cunningham, A. & Huszár, F. Lossy image compression with compressive autoencoders. Preprint at arXiv https://arxiv.org/abs/1703.00395 (2017).
15. Duan X Liu J Zhang E Efficient image encryption and compression based on a VAE generative model J. Real.-Time Image Process. 2019 16 765 773 10.1007/s11554-018-0826-4
Duan, X., Liu, J. & Zhang, E. Efficient image encryption and compression based on a VAE generative model. J. Real.-Time Image Process. 16, 765–773 (2019).10.1007/s11554-018-0826-4
16. Agustsson, E., Tschannen, M., Mentzer, F., Timofte, R. & Gool, L. V. in Proceedings of the IEEE/CVF International Conference on Computer Vision 221-231 (2019).
17. Sitzmann V Martel J Bergman A Lindell D Wetzstein G Implicit neural representations with periodic activation functions Adv. neural Inf. Process. Syst. 2020 33 7462 7473
Sitzmann, V., Martel, J., Bergman, A., Lindell, D. & Wetzstein, G. Implicit neural representations with periodic activation functions. Adv. neural Inf. Process. Syst. 33, 7462–7473 (2020).
18. Mildenhall B Nerf: Representing scenes as neural radiance fields for view synthesis Commun. ACM 2021 65 99 106 10.1145/3503250
Mildenhall, B. et al. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM 65, 99–106 (2021).10.1145/3503250
19. Dupont, E., Goliński, A., Alizadeh, M., Teh, Y. W. & Doucet, A. Coin: compression with implicit neural representations. Preprint at arXiv https://arxiv.org/abs/2103.03123 (2021).
20. Chen H Nerv: Neural representations for videos Adv. Neural Inf. Process. Syst. 2021 34 21557 21568
Chen, H. et al. Nerv: Neural representations for videos. Adv. Neural Inf. Process. Syst. 34, 21557–21568 (2021).
21. Gallant in IEEE 1988 International Conference on Neural Networks 657-664 vol. 651 (IEEE, 1988).
22. Genova, K., Cole, F., Sud, A., Sarna, A. & Funkhouser, T. Local deep implicit functions for 3D shape. Preprint at arXiv https://arxiv.org/abs/1912.06126 (2019).
23. Strümpler, Y., Postels, J., Yang, R., Gool, L. V. & Tombari, F. in European Conference on Computer Vision 74-91 (Springer, 2022).
24. Yang, R. et al. Sharing Massive Biomedical Data at Magnitudes Lower Bandwidth Using Implicit Neural Function. bioRxiv, 2022.2012. 2003.518948 (2022).
25. Dupont, E. et al. COIN++: Neural compression across modalities. Preprint at arXiv https://arxiv.org/abs/2201.12904 (2022).
26. Sheibanifard A Yu H A Novel Implicit Neural Representation for Volume Data Appl. Sci. 2023 13 3242 10.3390/app13053242
Sheibanifard, A. & Yu, H. A Novel Implicit Neural Representation for Volume Data. Appl. Sci. 13, 3242 (2023).10.3390/app13053242
27. Liang, R., Sun, H. & Vijaykumar, N. Coordx: Accelerating implicit neural representation with a split mlp architecture. Preprint at arXiv https://arxiv.org/abs/2201.12425 (2022).
28. Xu, J. et al. NeSVoR: Implicit Neural Representation for Slice-to-Volume Reconstruction in MRI. IEEE Transactions on Medical Imaging (2023).
29. Van der Maaten, L. & Hinton, G. Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 11 (2008).
30. Peng, W.-J., Chen, C.-W., Lin, Y.-H. & Chen, M.-F. in 2022 IEEE International Conference on Imaging Systems and Techniques (IST) 1-6 (IEEE, 2022).
31. Chilamkurthy, S. et al. Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study. 392, 2388-2396 [Dataset]. 10.48550/arXiv.1803.05854 (2018).
32. Fonseca, C. G. et al. The Cardiac Atlas Project—an imaging database for computational modeling and statistical atlases of the heart. Bioinformatics 27, 2288-2295 [Dataset].10.1093/bioinformatics/btr360 (2011).
33. Matuszewski, D. & Sintorn, I.-M. TEM virus dataset. Mendeley Data 3, x4dwwfwtw3[Dataset]. 10.17632/x4dwwfwtw3.1 (2021).
34. Stringer C Wang T Michaelos M Pachitariu M Cellpose: a generalist algorithm for cellular segmentation Nat. Methods 2021 18 100 106 10.1038/s41592-020-01018-x 33318659
Stringer, C., Wang, T., Michaelos, M. & Pachitariu, M. Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021).33318659 10.1038/s41592-020-01018-x
35. Patriarchi T Ultrafast neuronal imaging of dopamine dynamics with designed genetically encoded sensors Science 2018 360 eaat4422 10.1126/science.aat4422 29853555
Patriarchi, T. et al. Ultrafast neuronal imaging of dopamine dynamics with designed genetically encoded sensors. Science 360, eaat4422 (2018).29853555 10.1126/science.aat4422
36. Yasuda R Imaging spatiotemporal dynamics of neuronal signaling using fluorescence resonance energy transfer and fluorescence lifetime imaging microscopy Curr. Opin. Neurobiol. 2006 16 551 561 10.1016/j.conb.2006.08.012 16971112
Yasuda, R. Imaging spatiotemporal dynamics of neuronal signaling using fluorescence resonance energy transfer and fluorescence lifetime imaging microscopy. Curr. Opin. Neurobiol. 16, 551–561 (2006).16971112 10.1016/j.conb.2006.08.012
37. Hospedales T Antoniou A Micaelli P Storkey A Meta-learning in neural networks: A survey IEEE Trans. pattern Anal. Mach. Intell. 2021 44 5149 5169
Hospedales, T., Antoniou, A., Micaelli, P. & Storkey, A. Meta-learning in neural networks: A survey. IEEE Trans. pattern Anal. Mach. Intell. 44, 5149–5169 (2021).
38. Chen, Y. & Wang, X. in European Conference on Computer Vision 170-187 (Springer, 2022).
39. Selvaraju, R. R. et al. in Proceedings of the IEEE international conference on computer vision 618-626 (2017).
40. Prakash, A., Moran, N., Garber, S., DiLillo, A. & Storer, J. in 2017 Data Compression Conference (DCC) 250-259 (IEEE, 2017).
41. Griffin, G., Holub, A. & Perona, P. Caltech-256 object category dataset. (2007).
42. Boergens KM webKnossos: efficient online 3D data annotation for connectomics Nat. Methods 2017 14 691 694 10.1038/nmeth.4331 28604722
Boergens, K. M. et al. webKnossos: efficient online 3D data annotation for connectomics. Nat. Methods 14, 691–694 (2017).28604722 10.1038/nmeth.4331
43. Wolny A Accurate and versatile 3D segmentation of plant tissues at cellular resolution Elife 2020 9 e57613 10.7554/eLife.57613 32723478
Wolny, A. et al. Accurate and versatile 3D segmentation of plant tissues at cellular resolution. Elife 9, e57613 (2020).32723478 10.7554/eLife.57613
44. Gustafsson N Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations Nat. Commun. 2016 7 12471 10.1038/ncomms12471 27514992
Gustafsson, N. et al. Fast live-cell conventional fluorophore nanoscopy with ImageJ through super-resolution radial fluctuations. Nat. Commun. 7, 12471 (2016).27514992 10.1038/ncomms12471
45. Wang Z Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning Nat. Methods 2021 18 551 556 10.1038/s41592-021-01058-x 33574612
Wang, Z. et al. Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning. Nat. Methods 18, 551–556 (2021).33574612 10.1038/s41592-021-01058-x
46. Zhao Y Isotropic super-resolution light-sheet microscopy of dynamic intracellular structures at subsecond timescales Nat. Methods 2022 19 359 369 10.1038/s41592-022-01395-5 35277709
Zhao, Y. et al. Isotropic super-resolution light-sheet microscopy of dynamic intracellular structures at subsecond timescales. Nat. Methods 19, 359–369 (2022).35277709 10.1038/s41592-022-01395-5
47. Ma, Y. & Yi, C. FMM Code. Zenodo10.5281/zenodo.13328988 (2024).
