
==== Front
Heliyon
Heliyon
Heliyon
2405-8440
Elsevier

S2405-8440(24)13303-8
10.1016/j.heliyon.2024.e37272
e37272
Research Article
Nonrigid registration method for longitudinal chest CT images in COVID-19
Iwao Yuma yuma1360@gmail.com
ab⁎
Kawata Naoko chumito_03@yahoo.co.jp
cde
Sekiguchi Yuki y_sekiguchi@chiba-u.jp
d
Haneishi Hideaki haneishi@faculty.chiba-u.jp
a
a Center for Frontier Medical Engineering, Chiba University, 1-33, Yayoi-cho, Inage-ku, Chiba-shi, Chiba, 263-8522, Japan
b Institute for Quantum Medical Science, National Institutes for Quantum Science and Technology, 4-9-1, Anagawa, Inage-ku, Chiba-shi, Chiba, 263-8555, Japan
c Department of Respirology, Graduate School of Medicine, Chiba University, 1-8-1, Inohana, Chuo-ku, Chiba-shii, Chiba, 260-8677, Japan
d Graduate School of Science and Engineering, Chiba University, Chiba, 263-8522, Japan
e Medical Mycology Research Center (MMRC), Chiba University, Japan
⁎ Corresponding authors. Center for Frontier Medical Engineering, Chiba University, 1-33, Yayoi-cho, Inage-ku, Chiba-shi, Chiba, 263-8522, Japan. yuma1360@gmail.com
31 8 2024
15 9 2024
31 8 2024
10 17 e3727215 3 2023
22 8 2024
30 8 2024
© 2024 The Authors
2024
https://creativecommons.org/licenses/by-nc/4.0/ This is an open access article under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).
Rationale and objectives

To analyze morphological changes in patients with COVID-19-associated pneumonia over time, a nonrigid registration technique is required that reduces differences in respiratory phase and imaging position and does not excessively deform the lesion region. A nonrigid registration method using deep learning was applied for lung field alignment, and its practicality was verified through quantitative evaluation, such as image similarity of whole lung region and image similarity of lesion region, as well as visual evaluation by a physician.

Materials and methods

First, the lung field positions and sizes of the first and second CT images were roughly matched using a classical registration method based on iterative calculations as a preprocessing step. Then, voxel-by-voxel transformation was performed using VoxelMorph, a nonrigid deep learning registration method. As an objective evaluation, the similarity of the images was calculated. To evaluate the invariance of image features in the lesion site, primary statistics and 3D shape features were calculated and statistically analyzed. Furthermore, as a subjective evaluation, the similarity of images and whether nonrigid transformation caused unnatural changes in the shape and size of the lesion region were visually evaluated by a pulmonologist.

Results

The proposed method was applied to 509 patient data points with high image similarity. The variances in histogram characteristics before and after image deformation were confirmed. Visual evaluation confirmed the agreement between the shape and internal structure of the lung field and the natural deformation of the lesion region.

Conclusion

The developed nonrigid registration method was shown to be effective for quantitative time series analysis of the lungs.

Keywords

Coronavirus disease 2019 (COVID-19)
Chest computed tomography (CT)
Nonrigid registration
Deep learning
==== Body
pmcAbbreviation list

Chronic obstructive pulmonary disease COPD

Coronavirus disease 2019 COVID-19

Computed tomography CT

Extracorporeal membrane oxygenation ECMO

High-flow nasal cannula HFNC

Peak signal-to-noise ratio PSNR

Reverse transcription polymerase chain reaction RT‒PCR

Structural similarity SSIM

Zero mean normalized cross-correlation ZNCC

1 Introduction

Coronavirus disease 2019 (COVID-19) has infected more than 670 million people worldwide as of January 2023, and the number of deaths has reached 6.8 million, with no signs of abatement [1]. Reverse transcription polymerase chain reaction (RT‒PCR) tests [2,3] are used to diagnose the presence of COVID-19 infection, but their sensitivity and specificity vary depending on the timing and technique of the test, and their inability to diagnose the presence of COVID-19-related pneumonia is problematic. On the other hand, chest computed tomography (CT) has been reported to have a high diagnostic rate for COVID-19-related pneumonia, exceeding 80 % [4], and many previous reports have described the usefulness of CT and its combination with deep learning to improve the pneumonia diagnosis rate [5].

However, most of these reports used cross-sectional CT and clinical data in the early stages of infection, and there have been few reports of COVID-associated pneumonia using longitudinal data. In other respiratory diseases, for example, chronic progressive respiratory diseases such as chronic obstructive pulmonary disease (COPD) and idiopathic lung fibrosis, image analysis using CT over time has been reported, including the relationship between morphological changes over time (such as emphysema, airway lesions, vascular lesions, and interstitial lesions) and physiological changes and prognosis [6,7], and the potential utility of longitudinal analysis in COVID-19 has also been presented [8].

To analyze changes over time in the CT image of the lung field, linear transformation such as rigid-body transformation and affine transformation is not sufficient. Instead, nonrigid registration processing is required to account for differences in inspiratory volume and posture during examination. This is a very important process not only for quantitative comparative analysis of images on a voxel-by-voxel basis but also for visual diagnosis by physicians in which images are displayed on a slice-by-slice basis and the same slice needs to be provided quickly and accurately. Nonrigid registration methods are an important preprocessing step in various medical image processing tasks and have been widely studied [[9], [10], [11]]. However, the number of parameters to be obtained is large, and classical methods require considerable processing time. In recent years, many methods using deep learning have been proposed [[12], [13], [14], [15], [16], [17]], enabling highly accurate positioning in a few seconds. In particular, VoxelMorph [12] is an unsupervised learning method that transforms a pair of images to be aligned into a voxel translation vector map, enabling fast and accurate alignment. The code is available on GitHub [18], making it easy for radiologists and researchers to test it.

Based on the above-mentioned advantages, we use VoxelMorph for image registration between two CT images of COVID-19 patients scanned during hospitalization. However, the direct registration using VoxelMorph is prone to generate unexpected distortion in the image. VoxelMorph is a method for obtaining local deformations, and for highly accurate registration, a proper linear translation must be performed as preprocessing to align the global position. In fact, the authors who developed VoxelMorph assumed that two images to be registered are affinely aligned as a preprocessing [12]. Therefore, following the original paper of VoxelMorph, we employ a two-step registration: affine transformation to align the rough lung field shape and position and VoxelMorph-based transformation to align the detailed structures.

It should be evaluated both objectively and subjectively if the proposed approach generates reasonably deformed images. CT images of COVID-19 patients over time are predicted to show large differences in the distribution of lesions in the lung fields, and the effect of such differences on alignment accuracy has not been clarified. The registration process should register the outer shape of the lung, bronchi and main vascular structure inside the lung between the first and second CT images. On the other hand, the fine structure should not be deformed too much in this process, especially in the lesion region. It is possible that the size and shape of a lesion will be highly different between the first and second CT images as the disease progresses. Such disease-oriented changes in lesions should remain. Therefore, it is necessary to objectively evaluate two factors: (1) the whole image similarity after alignment and (2) the invariance of the lesion region. The whole image similarity was evaluated by four different indices: zero mean normalized cross-correlation (ZNCC), structural similarity (SSIM), peak signal-to-noise ratio (PSNR), and dice coefficient. For lesion invariance, we evaluated the similarity of the lesion region, specifically by a number of first-order statistics calculated from histograms of voxel intensities, and 3D shape features of lesion region in the lung. Finally, to subjectively verify whether the transformed lesion region shows clinically practical performance, a visual evaluation of the shape concordance and lesion region constancy was performed by a pulmonologist on a set of randomly selected samples.

2 Materials and methods

2.1 Study subjects

We enrolled 819 consecutive COVID-19 patients who were hospitalized and treated at Aoba Municipal Hospital from February 2020 to September 2021. The inclusion criterion for this retrospective study was patients with symptoms suspicious for COVID-19 who were diagnosed with COVID-19 during the COVID-19 outbreak. The patients underwent RT‒PCR tests of nasopharyngeal swab samples to establish a COVID-19 diagnosis. Patients with a positive PCR test result and a request for treatment and hospitalization were enrolled. During hospitalization, a first CT scan was performed at admission except for some patients, and a second CT scan was performed to follow the progression of pneumonia, including exacerbations. After excluding subjects under 20 years of age (n = 31), subjects who did not undergo the first CT scan (n = 32), subjects with data mismatches (n = 14), pregnant individuals (n = 3), transferred patients (n = 1), and subjects who did not undergo the second CT scan (n = 229), 509 patients were enrolled (Fig. 1).Fig. 1 Flowchart of the patient population.

Note: COVID-19 = coronavirus disease 2019. Of the 819 participants, 81 were excluded because of the following reasons: age <20 years (n = 31), pregnancy (n = 3), no CT scans (n = 32), hospital transfer (n = 1), and data mismatch (n = 14). Another 229 patients were excluded because they had not undergone longitudinal scans.

Fig. 1

This retrospective multicenter study was approved by the Institutional Review Boards of Chiba University (No. 4074) and Aoba Municipal Hospital (No. 20200301). The requirement for written informed consent was waived. To avoid any potential breach of patient confidentiality, the data were deidentified and had no linkage to the researchers.

2.2 Details of CT image data

The details of the first and second CT images used in this study are as follows. At the first CT scan, the patients underwent chest CT using 80-row CT scanners (Siemens, Erlangen, Germany) and were scanned from the thoracic inlet to the diaphragm during full inspiration without contrast enhancement. The CT settings were as follows: 120 kV, CT automatic exposure control, and a gantry rotation time of 0.5s. All images were reconstructed using a soft reconstruction kernel (I40f) and a sharp reconstruction kernel (B70f) with a slice thickness of 3 mm and a reconstruction interval of 3 mm. At the second CT scan, the patients underwent chest CT using 64-row CT scanners (Aquilion Precision, Canon Medical Systems, Otawara, Tochigi, Japan). The CT settings on the second scan were the same as those on the first scan. All images were reconstructed using a soft reconstruction kernel (FC13) and a sharp reconstruction kernel (FC53) with a slice thickness of 3 mm and a reconstruction interval of 3 mm.

2.3 Registration method

In the proposed process, the first and second CT images of the same patient were used as inputs. Because the effect of scanner differences appeared in the high-frequency component between the first and the second CT images, a Gaussian filter (σ = 1.0) was applied as a preprocessing step to reduce the differences in CT scanners. The images were then cropped with a rectangle around the lung fields and resized to an image size of 256 × 256× 128 voxels. As mentioned in the introduction, two-step registration technique was applied to the second CT image. As mentioned in the introduction, two types of registration techniques were considered for the second CT images as a preprocess. One was rigid body transformation, which derives 6 parameters consisting of rotation and translation, and the other was affine transformation, which derives 15 parameters including scale change and shear deformation. Since the lung field is a region that is scaled and deformed by breathing, we chose the affine transformation. VoxelMorph was then applied to the output of the affine transformation. The details of each process are as follows.

2.3.1 3D affine transformation

A 3D affine transformation was performed to match the differences in posture and rough lung field shape during imaging. The 3D translations are represented as tx,ty,andtz; the rotations about each axis are represented as rx,ry,andrz; the scaling about each axis is represented as sx,sy,andsz; and the 6 parameters responsible for the 3D shear deformation are represented as hxy,hyx,hxz,hzx,hyz,andhzy. A total of 15 parameters were estimated. If the 3D coordinates before deformation are expressed as (i,j,k,1) using homogeneous coordinates, the coordinates after deformation (i´,j´,k´,1) are expressed as follows using the product of homogeneous transformation matrices:(1) (i´j´k´1)=(1hyxhzx0hxy1hzy0hxzhyz100001)(sx0000sy0000sz00001)(10000cosrx−sinrx00sinrxcosrx00001)(cosry0−sinry00100sinry0cosry00001)(cosrz−sinrz00sinrzcosrz0000100001)(100tx010ty001tz0001)(ijk1)

The Nelder‒Mead method [19], which does not require the derivation of derivatives, was used to estimate the set of parameters that maximized the amount of mutual information [20] between the first and second CT images transformed using Equation (1). The above positioning method was implemented in C++.

2.3.2 Nonrigid registration by VoxelMorph

The network architecture of VoxelMorph is shown in Fig. 2. The 3D-UNet structure was used to transform an input pair of first CT image and second CT image after affine transformation into a 3D deformation vector map. Nonrigid registration was completed by applying this vector map shift to each voxel in the second CT image. In addition to image similarity, the complexity of the transformation vector was used as the loss to enforce a smooth deformation. Unsupervised learning with VoxelMorph was performed on pairs of first CT images and second CT images with 3D affine transformations. VoxelMorph is a framework uploaded to GitHub by the original developer [17].Fig. 2 Network architecture for VoxelMorph.

Fig. 2

2.3.3 Experimental settings

Of the 509 datasets, 409 were used for training, and the remaining 100 were used for accuracy tests. In VoxelMorph training, loss was defined by a weighted sum of cross-correlation and gradient loss. Preliminary experimental validation showed that setting the weights to 1.0, based on the optimal value proposed by Balakrishnan G et al. [12], provided the best balance between accuracy and stability. The training epoch was 120 using a workstation equipped with a GeForce RTX3090 (NVIDIA Corp., Santa Clara, CA, USA).

The proposed process was followed to deform the lungs nonrigidly, and the lung field shape in the second CT image was registered to the first CT image.

2.4 Evaluation

The learned parameters were applied to the evaluation data for accuracy verification. The Gaussian filter applied as a preprocessing step to the training data was not applied to the evaluation data to preserve imaging features.

As mentioned earlier, we evaluated the image registration result in both objective and subjective ways. The details of the evaluation method are described in this subsection. Fig. 3 shows the scheme of the objective evaluation. Note that while the whole image similarity is evaluated between the first CT (Fig. 3 (a)) and the second CT image after deformation (Fig. 3 (c, d)), the lesion similarity is evaluated between the original second CT image (Fig. 3 (b)) and the second CT image after deformation (Fig. 3 (d)).Fig. 3 Objective evaluation scheme.

Note: First CT images (a) and second CT images (c) and (d) were compared.The second CT images were generated from the original image (b), an image with affine transformation (c), and an image with affine transformation + VoxelMorph (d).Comparisons were made with the first CT images.

Fig. 3

2.4.1 Whole image similarity

In Fig. 3, the solid line indicates the similarity of the affine-transformed second CT image (Fig. 3 (c)) and the VoxelMorph-transformed second CT image (Fig. 3 (d)) to the first CT image (Fig. 3 (a)). The ZNCC, PSNR, and SSIM values were derived as the pixel similarity. In addition, pulmonary vascular regions were extracted using −720 HU as the threshold [21], and anatomical shape similarity was calculated using dice coefficients.

2.4.2 Invariance of the lesion region

For the analysis, the area above −500 HU in the inner lung field was extracted by thresholding as a high-attenuation area and used as a lesion mask [22]. This mask was extracted from the original 2nd CT image (Fig. 3 (b)) and the 2nd CT image after applying affine + VoxelMorph (Fig. 3 (d)). To confirm the invariance of the lesion regions, the first-order statistics calculated from the histogram of voxel intensities and the 3D shape features of the extracted lesion regions were measured using Pyradiomics (dotted line in Fig. 3). If the respiratory phase is significantly different, the CT values of the lung parenchyma region will change, resulting in a change in the histogram distribution. However, the COVID-19 CT images used in this study were all taken in the maximal inspiratory state, and it was assumed that such effects were minimal.

The Python package Pyradiomics [23] was used to extract image features from the lesion region within the original second CT images and the second CT images after nonrigid registration, and the Wilcoxon signed-rank test was performed for each feature to see if statistically significant differences occurred.

2.4.3 Visual evaluation by a pulmonologist

Scoring of emphysema and interstitial pneumonia has been reported for the quantitative assessment of observers. Referring to previous reports [24], in the present study, an evaluation was performed on the agreement rate of some typical structures in the lung field.

Visual evaluation by a pulmonologist (NK experience>20 years) was performed on the data of 50 randomly selected cases for evaluation. Two types of evaluation items were used. One was shape congruency, which was assessed in three cross sections of the lower pulmonary vein inlet, tracheal branch, and upper end of the arch, which are commonly used in the evaluation of clinical images.

The specific evaluation items were whether (1) the running direction and branching of pulmonary vessels and (2) the position of bronchi in the lung field were consistent within the cross section on the first and second CT images. The areas of congruence were evaluated at four levels: less than 20 %, 20 %–50 %, 50 %–80 %, and 80 % or higher.

The other evaluation item was whether there was any significant deformation in the size, shape or texture of the lesion, which indicated COVID-19-related pneumonia, between the original second CT image and the registered second CT image within the same three cross sections. If the morphing process causes excessive changes in the size, shape or texture of the lesion, it will have a significant impact on diagnostic accuracy, which is undesirable. The visual agreement was evaluated at four levels: less than 20 % (disagree), 20 %–50 %, 50 %–80 %, and 80 % or higher (agree).

3 Results

The demographics of the participants are shown in Table 1. The average patient age was 54 years, males predominated, and the average time interval from symptom onset to the first CT scan was 5.8 days and from onset to the second CT scan was 8.8 days. The number of patients receiving oxygen was 213 (41.9 %), and 24 (4.7 %) were on oxygenation at the time of admission. The number of patients with high-flow nasal cannula (HFNC) therapy was 68 (13.4 %). Eighteen patients (3.5 %) were intubated. Two patients needed extracorporeal membrane oxygenation (ECMO). Thirteen patients did not survive.Table 1 Patient characteristics and disease severity.

Table 1	(n = 509)	
Age	54 (20–97)	
Gender (male)	333 (65.4 %)	
BMI	25.2 ± 4.4	
Symptom onset to Admission CT (days)	5.8 ± 3.2	
Symptom onset to discharge CT (days)	14.6 ± 6.1	
Disease Severity	
oxygen supplementation	213 (41.9 %)	
oxygen supplementation	24 (4.7 %)	
at admission	
High-flow nasal oxygen	68 (13.4 %)	
intubation	18 (3.5 %)	
ECMO	2 (0.0/39 %)	
Survival/death	496 (97.4 %)/13 (2.6 %)	
Note: BMI = body mass index, ECMO = extracorporeal membrane oxygenation.

3.1 Nonrigid registration

An example of the results of applying nonrigid deformation is shown in Fig. 4. Panels (a)-(d) show slices with identical image coordinates. In Fig. 4 (a) and 4 (b), which are original CT pairs, there are significant differences in the position, external shape, and internal structure of the lungs. In Fig. 4 (c), the affine transformation improves the rough shape of the lungs in each axial, coronal, and sagittal slice, as well as the similarity of their positions in the images. Nevertheless, there are some differences in the vessel direction in the right lung in Axial 2, bronchial shape in the coronal slice, and vessel direction in the lower right lung (sagittal). In Fig. 4 (d), it is shown that VoxelMorph greatly improves the similarity of microstructures within the lung field as described above. Furthermore, in some cases, such as the lower right lung in Axial 3 of Fig. 4, lesions that were not present on the first CT image but are present on the second CT image may be seen due to progression of the disease stage. Note that such areas are not lost due to over-alignment with the first CT, but remain after registration.Fig. 4 The same-level slices for the upper end of the arch (Axial 1), trachea (Axial 2), and lower pulmonary vein inlet (Axial 3) are shown ((a)–(b)). The tracheal branch (coronal) and right lung center (sagittal) of the aorta in the first CT image are shown for each image ((b)–(d)).

Note: First CT image (a); second CT image (b); affine alignment result of second CT image (c); VoxelMorph applied to second CT image (d).

Fig. 4

3.2 Evaluation of image similarity

The similarity between the first CT images and the second CT images with affine transformation and with VoxelMorph were evaluated using four indices: ZNCC (Fig. 5 (a)), SSIM (Fig. 5 (b)), PSNR (Fig. 5 (c)), and dice coefficient (Fig. 5 (d)). All indices were significantly superior to that when only affine transformations were applied.Fig. 5 Image similarity evaluation results between the first CT image and affine alignment or affine + VoxelMorph results. Comparison of zero mean normalized cross-correlation (a), comparison of structural similarity (b), comparison of the peak signal-to-noise ratio (c), and dice coefficient of pulmonary vascular region (d).

Fig. 5

3.3 Verification of lesion region invariance

Fig. 6 lists the first-order statistics and 3D shape features of lesion region obtained from the second CT images before and after the nonrigid registration, and the p values in the Wilcoxon signed-rank test were used to evaluate significant differences. First-order statistics refers to 19 features derived from statistics of voxel intensities in an image, such as the maximum, minimum, median, and entropy. 3D shape features include volume, maximum length, sphericity, etc.Fig. 6 P value summary of the Wilcoxon signed-rank test for first order statistics and 3D shape features before and after nonrigid registration in the lung lesion region.

Fig. 6

No significant differences were confirmed, as all of the features had p values well above 0.05, which is the criterion for determining significant differences. This suggests that the characteristics of the lesion region did not change significantly before and after nonrigid registration.

3.4 Visual evaluation by a pulmonologist

Fig. 7 shows the results of visual evaluation by a pulmonologist.Fig. 7 Summary of visual evaluation results in the three axial level slices.

Similarity of the structure (a). Similarity of lesion region shape, size, and texture of COVID-19 pneumonia (b).

Fig. 7

Approximately 80 % of the data for both shape (Fig. 7 (a)) and lesion similarity (Fig. 7 (b)) showed the best performance in all three slices evaluated, and the other slices showed the next best performance.

The percentage of a score of 3 (agreement of 80 % or higher) for agreement of lung structure and pneumonia exceeded 90 %. The percentage of a score of 2 (agreement between 50 and 80 %) was less than 10 %.

The causes of a score of 2 included individuals with poor breath-holding (n = 1) and individuals with severe pneumonia and considerably low lung field permeability (n = 1).

4 Discussion

The processing developed in the present study enabled highly accurate nonrigid registration in lung fields with various deformations. Visual evaluation clearly showed that the lung structures, such as the distribution of pulmonary vessels and bronchi, were well matched.

The median values for each quantitative evaluation were 0.87 for ZNCC, 0.80 for SSIM, and 22 dB for the PSNR, all showing very high similarity. The improvement in similarity from the affine-transformed state to nonrigid registration was also visually and statistically evident. On the other hand, several results showed values as low as 0.6 for ZNCC, 0.5 for SSIM, and 13 for the PSNR. A typical example is shown in Fig. 8. As Fig. 8 (a) and 8 (b) show, one of the reasons for the decrease in image similarity was the increase in the number of lesion sites due to the progression of the disease stage. The increase in lesions can be observed in Fig. 8 (c) after affine transformation and Fig. 8 (d) with non-rigid deformation. Actually, the median value of the dice coefficient for anatomical similarity was also as low as 0.7. We speculated that the structure of pulmonary vessels obtained by the simple thresholding method might be useful for evaluating anatomical similarity. However, the simple thresholding method could hardly separate pulmonary vessels from lung field lesions. Due to the progression of the disease, it was difficult to clearly evaluate the similarity of pulmonary vessels in an objective way. Nevertheless, the similarity of the internal structures was subjectively confirmed by the fact that some of the structures were consistent with those in the first CT image, such as the interlobar cleft that can be seen in the upper right lung of Axial 2, the distribution of pulmonary vessels in the middle right lung of Axial 3, and the interlobar cleft in the coronal and sagittal slices. Interestingly, when this alignment process was applied to COVID-19 patients with different stages and lesions, there was no significant lesion loss or enhancement in visual assessment, although there was some change in shape. This was also evident from the analysis of changes in radiomics first-order features. None of the first-order features and 3D shape features showed a significant change in trend before or after registration.Fig. 8 Example of low-similarity results. The upper end of the arch (Axial 1), tracheal branch (Axial 2), lower pulmonary vein inlet (Axial 3), tracheal branch (coronal), and right lung center (sagittal) of the aorta in the first CT image are presented for each image. First CT image (a); second CT image (b); affine alignment result of second CT image (c); VoxelMorph application to second CT image (d).

Fig. 8

Our results indicate that the lung field nonrigid alignment process is capable of highly accurate alignment even in patients with COVID-19 pneumonia and retains the image features that are necessary for observation of changes over time. Since the beginning of the pandemic, numerous cross-sectional studies on CT analysis of COVID-19-related pneumonia have been reported [[25], [26], [27]]. There have also been several reports on morphologic changes over time [28,29]. Longitudinal morphological changes need to be elucidated for COVID-19. The proposed method will be useful for the quantitative assessment of those changes.

In COPD and idiopathic pulmonary fibrosis, morphological changes over time and their association with prognosis have been reported. The majority of longitudinal studies have analyzed lung parenchyma with conventional methods after confirming that total lung area or total lung volume did not change [30]. In COPD, nonrigid registration processing using inspiratory and expiratory images has been proposed for morphological and longitudinal analyses of disease progression [[31], [32], [33]]. When examining morphological changes over time, it is necessary to correct the changes in the images caused by inspiratory level and body position at the time of imaging.

The present method may be applicable to the image evaluation of these chronic progressive diseases and the assessment of disease progression. The application of CT images to multiple time phases containing lesion changes, such as in COVID-19-associated pneumonia, is likely to be more technically challenging than the application of CT images of inspiration and expiration in the same time phase. The data used in the present study included data with different respiratory dynamics, but visual assessment by an expert physician allowed registration without major problems. The maximum and average amount of transformation by VoxelMorph were about 31 mm and 2 mm, respectively. The small average value means that the whole region is roughly registered by the affine transformation. On the other hand, the maximum amount of transformation means that the VoxelMorph allows the local large movements in registration. In the post pandemic era, the proposed method may also be useful for longitudinal morphological analysis of chronic respiratory diseases.

This study had several limitations. First, this was a retrospective study. Our data were obtained from a relatively small number of participants recruited from a single institution. Second, data on treatment and vaccination for COVID-19 infection were not included in the current study. These preliminary results should be confirmed in a larger multicenter longitudinal cohort.

5 Conclusion

We applied VoxelMorph, a nonrigid registration method using deep learning, to time series chest CT images and demonstrated that it could achieve highly accurate registration. The results also showed that there was no significant change in image features before and after deformation, indicating that the method can be applied to time series analysis.

Data availability statement

The datasets used and/or analyzed during the current study are available from the corresponding author upon reasonable request.

Ethics statement

This retrospective study was performed in accordance with the Declaration of Helsinki and approved by the Institutional Review Boards of Chiba University (No. 4074) and Aoba Municipal Hospital (No. 20200301). The requirement for written informed consent was waived. To avoid any potential breach of patient confidentiality, the data were deidentified and had no linkage to the researchers.

CRediT authorship contribution statement

Yuma Iwao: Writing – review & editing, Writing – original draft, Visualization, Software, Data curation, Conceptualization. Naoko Kawata: Data curation. Yuki Sekiguchi: Software, Data curation. Hideaki Haneishi: Supervision.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

This research was partially supported by the Ministry of Education, Science, Sports and Culture, a Grant-in-Aid for Scientific Research (C) (22K12836), and a 10.13039/100019085 Japanese Respiratory Foundation (JRF) Grant. These funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

We would like to express our sincere gratitude to Dr. Matsuura, Dr. Nagayoshi, and Dr. Takiguchi of Chiba Aoba Municipal Hospital for their invaluable assistance in data collection.
==== Refs
References

1 Coronavirus Disease (COVID-19) Situation Reports, (n.d.). https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports (accessed February 8, 2023).
2 Suzuki T. Kutsuna S. Nakamura K. Ide S. Moriyama Y. Saito S. Morioka S. Ishikane M. Kinoshita N. Hayakawa K. Ohmagari N. Difficulty of downscaling the precautions for coronavirus disease-19 based on negative throat polymerase chain results in the early phase of infection J. Infect. Chemother. 26 2020 851 853 10.1016/J.JIAC.2020.05.002 32423700
3 Uwamino Y. Nagata M. Aoki W. Fujimori Y. Nakagawa T. Yokota H. Sakai-Tagawa Y. Iwatsuki-Horimoto K. Shiraki T. Uchida S. Uno S. Kabata H. Ikemura S. Kamata H. Ishii M. Fukunaga K. Kawaoka Y. Hasegawa N. Murata M. Accuracy and stability of saliva as a sample for reverse transcription PCR detection of SARS-CoV-2 J. Clin. Pathol. 74 2021 67 68 10.1136/JCLINPATH-2020-206972 32928941
4 Fang Y. Zhang H. Xie J. Lin M. Ying L. Pang P. Ji W. Sensitivity of chest CT for COVID-19: comparison to RT-PCR Radiology 296 2020 E115 E117 10.1148/RADIOL.2020200432 32073353
5 Wang H. Wang L. Lee E.H. Zheng J. Zhang W. Halabi S. Liu C. Deng K. Song J. Yeom K.W. Decoding COVID-19 pneumonia: comparison of deep learning and radiomics CT image signatures Eur. J. Nucl. Med. Mol. Imag. 48 2021 1478 1486 10.1007/S00259-020-05075-4
6 Jacob J. Bartholmai B.J. Rajagopalan S. Van Moorsel C.H.M. Van Es H.W. Van Beek F.T. Struik M.H.L. Kokosi M. Egashira R. Brun A.L. Nair A. Walsh S.L.F. Cross G. Barnett J. De Lauretis A. Judge E.P. Desai S. Karwoski R. Ourselin S. Renzoni E. Maher T.M. Altmann A. Wells A.U. Predicting outcomes in idiopathic pulmonary fibrosis using automated computed tomographic analysis Am. J. Respir. Crit. Care Med. 198 2018 767 776 10.1164/RCCM.201711-2174OC 29684284
7 Shima H. Tanabe N. Oguma A. Shimizu K. Kaji S. Terada K. Oguma T. Kubo T. Suzuki M. Makita H. Sato A. Nishimura M. Sato S. Konno S. Hirai T. Subtyping emphysematous COPD by respiratory volume change distributions on CT Thorax 2022 10.1136/thoraxjnl-2021-218288
8 Kawata N. Iwao Y. Matsuura Y. Suzuki M. Ema R. Sekiguchi Y. Sato H. Nishiyama A. Nagayoshi M. Takiguchi Y. Suzuki T. Haneishi H. Prediction of oxygen supplementation by a deep-learning model integrating clinical parameters and chest CT images in COVID-19 Jpn. J. Radiol. 41 2023 1359 1372 10.1007/s11604-023-01466-3 37440160
9 Klein A. Andersson J. Ardekani B.A. Ashburner J. Avants B. Chiang M.C. Christensen G.E. Collins D.L. Gee J. Hellier P. Song J.H. Jenkinson M. Lepage C. Rueckert D. Thompson P. Vercauteren T. Woods R.P. Mann J.J. Parsey R.V. Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration Neuroimage 46 2009 786 802 10.1016/J.NEUROIMAGE.2008.12.037 19195496
10 Avants B.B. Tustison N.J. Song G. Cook P.A. Klein A. Gee J.C. A reproducible evaluation of ANTs similarity metric performance in brain image registration Neuroimage 54 2011 2033 2044 10.1016/J.NEUROIMAGE.2010.09.025 20851191
11 Fischl B. FreeSurfer Neuroimage 62 2012 774 10.1016/J.NEUROIMAGE.2012.01.021 22248573
12 Balakrishnan G. Zhao A. Sabuncu M.R. Guttag J. Dalca A.V. VoxelMorph: a learning framework for deformable medical image registration IEEE Trans. Med. Imag. 38 2019 1788 1800 10.1109/TMI.2019.2897538
13 Lafarge M.W. Moeskops P. Veta M. Pluim J.P.W. Eppenhof K.A.J. Deformable image registration using convolutional neural networks 27 10.1117/12.2292443 2018
14 Rohé M.M. Datar M. Heimann T. Sermesant M. Pennec X. SVF-net: learning deformable image registration using shape matching Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 10433 LNCS 2017 266 274 10.1007/978-3-319-66182-7_31
15 Sokooti H. de Vos B. Berendsen F. Lelieveldt B.P.F. Išgum I. Staring M. Nonrigid image registration using multi-scale 3D convolutional neural networks Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 10433 LNCS 2017 232 239 10.1007/978-3-319-66182-7_27
16 de Vos B.D. Berendsen F.F. Viergever M.A. Sokooti H. Staring M. Išgum I. A deep learning framework for unsupervised affine and deformable image registration Med. Image Anal. 52 2018 128 143 10.1016/j.media.2018.11.010 30579222
17 Yang X. Kwitt R. Styner M. Niethammer M. Quicksilver: fast predictive image registration – a deep learning approach Neuroimage 158 2017 378 396 10.1016/J.NEUROIMAGE.2017.07.008 28705497
18 GitHub - voxelmorph/voxelmorph: Unsupervised Learning for Image Registration, (n.d.). https://github.com/voxelmorph/voxelmorph (accessed February 8, 2023).
19 Nelder J.A. Mead R. A simplex method for function minimization Comput. J. 7 1965 308 313 10.1093/comjnl/7.4.308
20 Viola P. Wells W.M. III Alignment by maximization of mutual information Int. J. Comput. Vis. 24 1997 137 154 10.1023/A:1007958904918
21 Matsuoka S. Washko G.R. Yamashiro T. Estepar R.S.J. Diaz A. Silverman E.K. Hoffman E. Fessler H.E. Criner G.J. Marchetti N. Scharf S.M. Martinez F.J. Reilly J.J. Hatabu H. Pulmonary hypertension and computed tomography measurement of small pulmonary vessels in severe emphysema Am. J. Respir. Crit. Care Med. 181 2010 218 225 10.1164/rccm.200908-1189OC 19875683
22 Klapsing P. Herrmann P. Quintel M. Moerer O. Automatic quantitative computed tomography segmentation and analysis of aerated lung volumes in acute respiratory distress syndrome—a comparative diagnostic study J. Crit. Care 42 2017 184 191 10.1016/j.jcrc.2016.11.001 28759880
23 pyradiomics, (n.d.). https://pyradiomics.readthedocs.io/en/latest/(accessed February 8, 2023).
24 Araiza A. Duran M. Patiño C. Marik P.E. Varon J. The Ichikado CT score as a prognostic tool for coronavirus disease 2019 pneumonia: a retrospective cohort study J Intensive Care 9 2021 10.1186/s40560-021-00566-4
25 Fang Y. Zhang H. Xie J. Lin M. Ying L. Pang P. Ji W. Sensitivity of chest CT for COVID-19: comparison to RT-PCR Radiology 296 2020 E115 E117 10.1148/radiol.2020200432 32073353
26 Colombi D. Bodini F.C. Petrini M. Maffi G. Morelli N. Milanese G. Silva M. Sverzellati N. Michieletti E. Well-aerated lung on admitting chest CT to predict adverse outcome in COVID-19 pneumonia Radiology 296 2020 E86 E96 10.1148/radiol.2020201433 32301647
27 Ai T. Yang Z. Hou H. Zhan C. Chen C. Lv W. Tao Q. Sun Z. Xia L. Correlation of chest CT and RT-PCR testing for coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases Radiology 296 2020 E32 E40 10.1148/radiol.2020200642 32101510
28 Bernheim A. Mei X. Huang M. Yang Y. Fayad Z.A. Zhang N. Diao K. Lin B. Zhu X. Li K. Li S. Shan H. Jacobi A. Chung M. Chest CT findings in coronavirus disease 2019 (COVID-19): relationship to duration of infection Radiology 295 2020 685 691 10.1148/radiol.2020200463
29 Pan F. Ye T. Sun P. Gui S. Liang B. Li L. Zheng D. Wang J. Hesketh R.L. Yang L. Zheng C. Time course of lung changes at chest CT during recovery from Coronavirus disease 2019 (COVID-19) Radiology 295 2020 715 721 10.1148/radiol.2020200370 32053470
30 Shin T. Kawata N. Tada Y. Ikari J. Matsuura Y. Matsuoka S. Matsushita S. Yanagawa N. Kasahara Y. Tatsumi K. Longitudinal changes in structural abnormalities using MDCT in COPD: do the CT measurements of airway wall thickness and small pulmonary vessels change in parallel with emphysematous progression? International Journal of COPD 12 2017 551 560 10.2147/COPD.S121405 28243075
31 Galbán C.J. Han M.K. Boes J.L. Chughtai K.A. Meyer C.R. Johnson T.D. Galbán S. Rehemtulla A. Kazerooni E.A. Martinez F.J. Ross B.D. Computed tomography-based biomarker provides unique signature for diagnosis of COPD phenotypes and disease progression Nat. Med. 18 2012 1711 1715 10.1038/nm.2971 23042237
32 Young A.L. Bragman F.J.S. Rangelov B. Han M.K. Galbán C.J. Lynch D.A. Hawkes D.J. Alexander D.C. Hurst J.R. Crapo J.D. Silverman E.K. Make B.J. Regan E.A. Beaty T. Begum F. Castaldi P.J. Cho M. DeMeo D.L. Boueiz A.R. Foreman M.G. Halper-Stromberg E. Hayden L.P. Hersh C.P. Hetmanski J. Hobbs B.D. Hokanson J.E. Laird N. Lange C. Lutz S.M. McDonald M.L. Parker M.M. Qiao D. Regan E.A. Silverman E.K. Wan E.S. Won S. Sakornsakolpat P. Prokopenko D. Al Qaisi M. Coxson H.O. Gray T. Hoffman E.A. Humphries S. Jacobson F.L. Judy P.F. Kazerooni E.A. Kluiber A. Lynch D.A. Newell J.D. Regan E.A. Ross J.C. Estepar R.S.J. Schroeder J. Sieren J. Stinson D. Stoel B.C. Tschirren J. Van Beek E. Van Ginneken B. Van Rikxoort E. Washko G. Wilson C.G. Jensen R. Everett D. Crooks J. Moore C. Strand M. Wilson C.G. Hokanson J.E. Hughes J. Kinney G. Lutz S.M. Pratte K. Young K.A. Bhatt S. Bon J. Make B.J. Martinez C. Murray S. Regan E.A. Soler X. Wilson C.G. Bowler R.P. Kechris K. Banaei-Kashani F. Curtis J.L. Martinez C.H. Pernicano P.G. Hanania N. Alapat P. Atik M. Bandi V. Boriek A. Guntupalli K. Guy E. Nachiappan A. Parulekar A. DeMeo D.L. Hersh C.P. Jacobson F.L. Washko G. Barr R.G. Austin J. D'Souza B. Pearson G.D.N. Rozenshtein A. Thomashow B. MacIntyre N. McAdams H.P. Washington L. McEvoy C. Tashjian J. Wise R. Brown R. Hansel N.N. Horton K. Lambert A. Putcha N. Casaburi R. Adami A. Budoff M. Fischer H. Porszasz J. Rossiter H. Stringer W. Sharafkhaneh A. Lan C. Wendt C. Bell B. Foreman M.G. Berkowitz E. Westney G. Bowler R.P. Lynch D.A. Rosiello R. Pace D. Criner G. Ciccolella D. Cordova F. Dass C. D'Alonzo G. Desai P. Jacobs M. Kelsen S. Kim V. Mamary A.J. Marchetti N. Satti A. Shenoy K. Steiner R.M. Swift A. Swift I. Vega-Sanchez M.E. Dransfield M. Bailey W. Bhatt S. Iyer A. Nath H. Wells J.M. Ramsdell J. Friedman P. Soler X. Yen A. Comellas A.P. Hoth K.F. Newell J.D. Thompson B. Kazerooni E.A. Martinez C.H. Billings J. Begnaud A. Allen T. Sciurba F. Bon J. Chandra D. Fuhrman C. Weissfeld J. Anzueto A. Adams S. Maselli-Caceres D. Ruiz M.E. Disease progression modeling in chronic obstructive pulmonary disease Am. J. Respir. Crit. Care Med. 201 2020 294 302 10.1164/rccm.201908-1600OC 31657634
33 Pu Y. Zhou X. Zhang D. Guan Y. Xia Y. Tu W. Lu Y. Zhang W. Fu C.C. Fang Q. de Bock G.H. Liu S. Fan L. Re-defining high risk COPD with parameter response mapping based on machine learning models International Journal of COPD 17 2022 2471 2483 10.2147/COPD.S369904 36217330
