
==== Front
Sci Rep
Sci Rep
Scientific Reports
2045-2322
Nature Publishing Group UK London

72469
10.1038/s41598-024-72469-x
Article
3D augmentation for volumetric whole heart segmentation
Sutassananon Krittanat 1
Kusakunniran Worapan worapan.kun@mahidol.ac.th

1
Orgun Mehmet 2
Siriapisith Thanongchai 3
1 https://ror.org/01znkr924 grid.10223.32 0000 0004 1937 0490 Faculty of Information and Communication Technology, Mahidol University, Nakhon Pathom, 73170 Thailand
2 https://ror.org/01sf06y89 grid.1004.5 0000 0001 2158 5405 School of Computing, Macquarie University, Macquarie Park, 2113 Australia
3 grid.10223.32 0000 0004 1937 0490 Department of Radiology, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok, 10700 Thailand
13 9 2024
13 9 2024
2024
14 2145913 5 2024
9 9 2024
© The Author(s) 2024
2024
https://creativecommons.org/licenses/by-nc-nd/4.0/ Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
Data augmentation is a technique usually deployed to mitigate the possible performance limitation from training a neural network model on a limited dataset, especially in the medical domain. This paper presents a study on effects of applying different rotation settings to augment cardiac volumes from the Multi-modality Whole Heart Segmentation dataset, in order to improve the segmentation performance. This study presents a comparison between conventional 2D (slice-wise) rotation primarily on the axial axis, 3D (volume-wise) rotation, and our proposed rotation setting that takes into account possible cardiac alignment according to its anatomy. The study has suggested two key considerations: 2D slice-wise rotation should be avoided when using 3D data for segmentation, due to intrinsic structural correlation between subsequent slices, and that 3D rotations may help improve segmentation performance on data previously unseen to the model.

Keywords

Data augmentation
Convolutional neural networks
Cardiac segmentation
Volumetric data
Subject terms

Computer science
Information technology
issue-copyright-statement© Springer Nature Limited 2024
==== Body
pmcIntroduction

Accurate measurement of cardiac substructures in medical images plays an important role in understanding their underlying anatomical properties, which help offer benefits in the diagnosis of cardiac diseases.

In practice, however, a precise measurement of the four cardiac chambers requires immense labour from medical practitioners to manually analyse each single slice from a cardiac image sequence, and aggregate results from those slices to form a volumetric measurement of the whole image sequence. For example, in order to produce the ground truth for the Multi-modality Whole Heart Segmentation Challenge (MM-WHS) dataset, it took trained medical professionals approximately six to ten hours to complete the annotation of a single cardiac volume1.

In recent years, the advances on neural network-based models for segmenting medical images have shown great potential for clinical applications. Neural networks normally rely heavily on both the quantity and the quality of the ground truths, on which networks could be trained on, and the lack of adequate data on any of the two dimensions could potentially pose an issue on model development. This is especially the case for segmenting medical images, where intensive resources are required to produce ground truth data and concerns like information privacy result in fewer training images available for training a segmentation model compared to other image segmentation domains.

To mitigate the impact of the limited training data, data augmentation techniques are usually deployed to artificially increase the amount of data exposed to the model. For medical images, this would mean applying either geometric transformations (e.g., rotation, scaling, shearing, zooming) or photometric transformations (e.g., blurring, sheering) on the currently available images, or a combination of both. In many cases, data augmentation can effectively alleviate performance issues when faced with limited data, but an adverse effect may be observed when the augmented data deviate too much from the statistical distribution of the population (i.e., abnormal or unnatural organ shapes), or fail to bring significant performance increase if they deviate too little from the original data.

In the 2018 MICCAI Left Atrium Segmentation Challenge, it has been shown that standard geometric augmentations helped improve the segmentation performance by 0.7%, whereas photometric transformations did not result in significant improvements2,3. It has also been observed that applying data augmentation on-the-fly yielded approximately similar results over and above using pre-augmented data alongside the original dataset.

Standard benchmarking segmentation models such as U-Net4, 3D U-Net5, and V-Net6 primarily deployed both geometric and photometric transformations. U-Net and 3D U-Net have deployed geometric transformations in forms of shifting, scaling, and rotating images together with smooth deformation via random placement vectors sampled from a Gaussian distribution. Instead of adaptively manipulating intensity values like in U-Net and 3D U-Net, V-Net applied histogram matching to adjust intensity values to match the distribution of another image randomly selected from the same dataset. Other prominent segmentation models including CE-Net7 have also deployed rotation and other geometric transformation with HSV colour jitting. AdaResU-Net8 has used image rotation in 90∘ range, while the others including CFUN9 and TransUNet10 have suggested augmentation as a mean to improve the model performance.

One notable case was the use of different ranges of rotation angles in nnU-Net11 alongside other augmentation strategies in both geometric and photometric category. Rotation angles were defined as up to 30∘ and 180∘ in both clockwise and anti-clockwise directions for 3D and 2D images, respectively. To prevent excessive augmentation with such expansive set of transformations, the authors introduced the probabilty of performing an augmentation: each transformation has a 16% chance of being applied, and chances of multiple transformation being applied together are multiplicative. That is, there was less chance of multiple transformation on one input compared to only one transformation at any given time.

For 3D medical image volume, for example cardiac CT scans from the MM-WHS dataset, each instance contains the structural information of an organ arranged into continuous slices that could be viewed from three different axes: axial, coronal, and sagittal. When traversing across an axis, spatial relationship between contiguous slices would provide the correlation how this structural information would change from one slice to another. In case of CT volumes, this spatial relationship helps regulate against abrupt changes between contiguous slices; abnormal changes would be an exception rather than a norm.

In case of segmenting these 3D cardiac volumes with 2D segmentation models like U-Net, these volumes would be parsed into the model one slice at a time; in other words, input to the segmentation models are made up of individual slices along the chosen primary axis of the volume. Standard on-the-fly augmentation on 2D slices would be calculated and applied on a per-slice basis, resulting in the interslice spatial relationship being not taken into account. In this case, applying a similar augmentation configuration across an individual training batch may help retain the spatial relationship. Unlike other geometric transformations like scaling and flipping, where the spatial relationship between neighbouring pixels are better preserved, applying rotation transformations however could result in relative larger deformation of this relationship. This is especially the case when applying rotations on a 3D volume in more than one axis. For example, applying a rotation on the coronal axis after a rotation on the axial axis would result in a fairly different alignment of cardiac substructures when viewed from a single slice of that volume.

To study the effect of applying different rotation settings on cardiac volumes, in regard to changes in segmentation performance of the neural network models, this works aim to provide a suggestion on further applications on related segmentation domains. In particular, we hypothesised that rotations settings that retain spatial relationship may help the model to perform better than those that does not actively seek to retain such relationship. We also hypothesised that taking into account possible cardiac alignments according to their underlying anatomical characteristics might also produce superior results, as this would prevent rotation results that would not be realistic and might mislead the segmentation model; alignments of cardiac substructures that we might never encounter in clinical cases.

Methods

To evaluate on the objectives, we have experimented with different rotation settings on 2D U-Net and 3D U-Net models, and measure their segmentation accuracy with Dice coefficient score. Experiments were performed in five-fold cross-validation sets to assess the degree of generalisability of the results.

Rotating a voxel

Linear geometric transformations—such as translation, scale, flip, and rotation—can be represented with numeric matrices. For example, given a cadiac voxel A of size x along the sagittal axis, y along the coronal axis, and z along the axial axis, we define rotation transforms by θ angle in each axis as follows:Raxial(θ)=cosθ-sinθ00sinθcosθ0000100001Rcoronal(θ)=cosθ0sinθ00100-sinθ0cosθ00001Rsagittal(θ)=10000cosθ-sinθ00sinθcosθ00001

Multiple transformations can be combined by taking a composite of transformation matrices. In general, composite transforms are non-commutative; that is, a change to the order of which transformations are applied would lead to different results. The exception for composite rotations are when these rotations share the same axis, or when there is a 180∘ rotation about perpendicular axes.

To preserve similar results regardless of any preferences on rotation order, we realigned axes of a volume in reference to the world axes after each rotation so that the axes does not change along as the result of the rotation operation. This was achieved in the implementation through successively replacing the original voxel with the rotated ones; Anew=R(A) instead of A′=R(A)

For the volumes from the MM-WHS dataset, the rotation axes are illustrated on Fig. 1. The sample rotation results are illustrated on Fig. 2, which simulates how the middle slice of a volume would change after the volume has been rotated in both the axial and coronal axes, and both in clockwise and anti-clockwise direction. Fig. 3 illustrates how this middle slices would be seen by a 2D segmentation model like U-Net. Coronal rotations may change how the middle slice of a sample volume would look like, as shown on Fig. 4Figure 1 Rotation axes for a sample volume from the MM-WHS dataset.

Figure 2 Changes to the middle slice of a sample volume from the MM-WHS dataset, after it was rotated with different settings.

Figure 3 The middle slice of a sample volume, after axial rotations with different rotation angles.

Figure 4 The middle slice of a sample volume, after coronal rotations with different rotation angles.

2D and 3D rotation settings

Different rotation settings were formulated to observe on the following objectives: (1) how an increase/decrease in angle ranges affects the segmentation accuracy, (2) whether there are significant differences between slice-wise (2D) and voxel-wise (3D) rotations, and (3) whether settings with a consideration on possible cardiac alignment could help improve the segmentation accuracy compared to those without.

Three sets of rotation angles were experimented: small (denoted with the superscript S) covered the range between 10∘ anti-clockwise to 10∘ clockwise, large (denoted with the superscript L) covered the range between 20∘ anti-clockwise to 20∘ clockwise, and proposed (denoted with the superscript P) covered the range between 0-30∘ clockwise in the axial axis and 10∘ anti-clockwise to 10∘ clockwise in the coronal axis. Each set was composed of different angles between the given range, with an increment of 5∘ between subsequent angles. For example, the small set contained the following angles: {-10,-5,0,5,10}, where zero angle signifies no rotation. The proposed set was experimental; the range of angles were determined by a radiologist particularly for the MM-WHS dataset. Sagittal rotations, according to the medical opinion in regard to the current dataset, may result in alignments that are not anatomically realistic, so they were not included in our proposed settings.

For 2D rotation, each individual slice in the same training batch would be rotated independently of any subsequent slices. This would replicate cases for other segmentation domains where subsequent 2D images may not necessarily contain any correlated information. For 3D rotation, each individual slice in the same training batch would receive the same rotation angle. In this way, the any correlation between subsequent slices may be kept intact, whereas rotating each individual slice differently may lead to abrupt changes in rotation angles between subsequent slices and might not warrant such correlation. For example, in 2D rotation with large settings, a slice might be rotated 20∘ in clockwise direction, and the subsequent slice might be rotated 20∘ in anti-clockwise direction.

To see whether our proposed set of rotation angles would outperform both small and large set on 2D rotations, we also experimented using only 0-30∘ on axial rotation in our comparison.

Segmentation models

The U-Net model, proposed by Ronneberger et al4, has been considered to be the benchmark for semantic segmentation models, especially in segmenting medical images. At its conception, U-Net outdated most conventional techniques that required comprehensive understanding of the image domain to manually design suitable segmentation methods. The architecture of U-Net mainly consists of two parts: the contracting path to capture and extract the image context, and the expanding path to provide accurate localisation of the context extracted. One key contribution to the success of U-Net is in its direct use of the captured context from the contracting path to help influence the localisation of the context in the expanding path through skip-forward linkage. Many adaptions of the original U-Net have been proposed and submitted to several medical image segmentation challenges including MM-WHS—and a number of them were the winning architectures in those challenges.

While U-Net generally achieves good performance in 2D images, its performance could not keep up with other dedicated 3D models in segmenting 3D volumes, where there exist correlation between subsequent slices within the volume. For the CT volumes in the MM-WHS dataset, it is observed to be structural constraints so that substantial variations in cardiac substructures between subsequent slices would be unlikely or not realistic. To adapt U-Net for 3D volumes, 3D U-Net5 was introduced with these following changes: (1) replacing 2D counterparts with 3D ones (i.e., 3D convolution kernels instead of 2D ones), (2) doubling the number of feature channels before max-pooling layers to avoid representation bottlenecks, and (3) using weighted cross-entropy loss to reduce emphasis on frequently seen background pixels. The increase in dimensionality of the 3D convolutional kernels help 3D U-Net to take into account information from both the preceeding the the subsequent slice, in addition to the spatial information from the current slice.

In our experiment, both U-Net and 3D U-Net were implemented according to the original architecture. U-Net has the network depth of 5, and starts with 64 features on the first level. 3D U-Net has the network depth of 4, and the first level first extracted 32 layers from then input before doubling up the channel t0 64. The only difference between our implementation and the original architecture was that we applied output padding, so that the output from a convolutional kernel retained the same dimension as the input.

Dataset

The Multi-modality Whole Heart Segmentation Challenge (MM-WHS)1 was launched in 2017 as a benchmark for whole heart segmentation algorithms on cardiac image sequences. The dataset provides 20 cardiac volumes in both the CT and MRI modalities. Each volume has 512 pixels in both the coronal and sagittal dimension, and between 177–363 pixels in the axial dimension.

In this study, however, we used all 20 voxels only from the CT modality. Figure 5 shows the middle slices of a sample voxel when viewed from different axes. Each cardiac chamber is annotated according to their positions on the sample slice.Figure 5 Annotated slices fom a sample voxel from the dataset. Left atrium (LA) is highlighted in red, right atrium (RA) in yellow, left ventricle (LV) in green, and right ventricle (RV) in blue, respectively.

Input pre-processing

Each volume was first resized and then normalised before entering the training and evaluation pipeline.

Firstly, each volume was resized to 64×128×128 along the axial, coronal, and sagittal dimension, respectively. There are two reasons behind this decision: to make volume size uniform across all volumes in the dataset, which is a prerequisite for both U-Net and 3D U-Net models, and that scaling the volume size down allows us to extend our experiment to cover more training batch sizes and relaxes hardware requirements for the training process. The chosen dimension sizes would provide a good balance to obtain useful insights while not compromising too much generalisability when volumes are scaled down too far from their original sizes.

Each volume was then normalised with a window level of 300 and a window width of 1000. Instead of normalising each volume by its own intensity range, or normalising with the dataset average, window normalising would be more robust against intensity variance both within a dataset and between datasets. This setting for window level and width has been determined by a medical opinion according to the characteristic of cardiac tissue and appropriate adjustments in regard to the dataset. As observed from Fig. 6, applying window adjustment resulted in better contrast between cardiac substructures and their surrounding areas.Figure 6 Effect of normalising and applying window adjustment on a sample voxel from the dataset.

Model training and evaluation

To evaluate the performance gain from applying rotations to complement 2D and 3D segmentation, U-Net4 and 3D U-Net5 were used. More advanced models with specialised components such as Attention U-Net12, MultiRes U-Net13, and TransUNet10 were not chosen due to our assumption that observations from those models might be less applicable for a general guideline for other segmentation models, compared to standard models like U-Net and 3D U-Net.

Each rotation settings was evaluated in five-fold cross-validation scheme, to simulate an overview on the generalisability of the setting. Dice coefficient was used to evaluate pixelwise similarity between the predicted segmentation of an input image and its corresponding ground truth labels. Soft Dice coefficient loss was derived from the standard Dice coefficient, to provide a continuous and differentiable gradient for model training.

For 2D segmentation models, training with a batch size of less than 4 were discarded since the model converged too slowly or not at all; their segmentation performance were significantly degraded comapred to those of higher batch sizes. For 3D segmentation models, we could only experiment up to a batch size of 4 due to hardware constraint. It is also noted that 3D segmentation models consume more resources compared to their 2D counterparts due to the increased dimension. Once the training was completed, the model was then reverted back to its best-performing state with the lowest Soft Dice coefficient loss and the having the highest Dice coefficient score. This was to ensure that the comparison between settings were made at their peak performance.

Regardless of the choice of segmentation models and rotation settings, the axial axis was the primary axis for segmentation. That is, the input volume could be rotated in any axes or any combination of axes, but it was only inputted to the model in the axial-dominant view as with contemporary segmentation models on cardiac substructures. This was designed to provide an impression on how our proposed combination of axial (0--30∘) and coronal (-10∘–10∘) rotation would perform in terms of segmenting from the axial view, and also how segmentation performance would change when the primary segmentation axis is different from the axis on which rotation was applied.

Adam optimiser14 was selected to dynamically adjust learning rates without requirements to fine tune across different models or different cross-validation folds to ensure good convergence. The learning rates for 2D U-Net and 3D U-Net were 1e-4 and 1e-2, respectively. All experiments were conducted on NVIDIA Tesla V100 GPUs.

Results

Tables 1 and  2 presents the summary of applying each rotation settings to U-Net and 3D U-Net, respectively. Detailed information supplementing both tables could be found in Tables 3 and  4. We only included results for U-Net starting from batch size of 32 in the summary table, as results for lower batch sizes are relatively poor compared to those trained with batch size of 32 or higher.

2D segmentation

As observed from Table 3, applying 2D augmentation resulted in inferior performance compared to both the baseline vanilla U-Net and other 3D augmentation settings. Increasing the batch size did not help improve the performance, in contrast to increasing the batch size in other rotation settings.

For smaller batch sizes (BS = 4, BS = 8, and BS = 16), results were inconclusive. There was no clear performance gain both in terms of training accuracy or testing accuracy.

For larger batch sizes (BS = 32, BS = 64, and BS = 128), the Dice coefficient scores for the baseline case suggested that the 2nd and the 4th fold may be more susceptible for the model to overfit the training data. As observed through the standard deviation values in all cases, U-Net produced more consistent results in the training data than in the testing data.

While the baseline model showed great performance in adapting to the training data across all folds, larger gaps between the training accuracy and the testing accuracy showed that the baseline model may tend to overfit more than those with 3D rotation settings. 3D rotation on the axial axis, despite showing lower training accuracy compared to the baseline results, exhibits better testing accuracy especially in larger batch sizes, though not significantly better. Results for coronal rotations are slightly inferior to axial rotations, and when combined with axial rotations in our mixed cases the results were inconsistent.Table 1 Average (mean and standard deviation) of Dice efficient score, for each of the rotation settings trained on U-Net.

Rotation setting	BS=32	BS=64	BS=128	
Train	Test	Train	Test	Train	Test	
Baseline	0.9959±0.0010	0.8759±0.0192	0.9932±0.0030	0.8762±0.0232	0.9909±0.0034	0.8771±0.0367	
2D AxialL	0.6207±0.0103	0.4931±0.0288	0.6225±0.0075	0.5030±0.0342	0.6194±0.0083	0.5036±0.0392	
AxialS	0.9775±0.0064	0.8767±0.0231	0.9772±0.0040	0.8791±0.0253	0.9759±0.0045	0.8937±0.0197	
AxialL	0.9670±0.0024	0.8802±0.0232	0.9698±0.0061	0.8794±0.0237	0.9665±0.0070	0.8928±0.0127	
AxialP	0.9724±0.0029	0.8798±0.0194	0.9721±0.0037	0.8863±0.0183	0.9740±0.0029	0.8966±0.0202	
CoronalL	0.9716±0.0040	0.8671±0.0400	0.9756±0.0027	0.8633±0.0321	0.9761±0.0025	0.8839±0.0311	
CoronalP	0.9805±0.0046	0.8703±0.0298	0.9793±0.0051	0.8765±0.0260	0.9774±0.0053	0.8758±0.0354	
MixedS	0.9796±0.0059	0.8648±0.0338	0.9826±0.0042	0.8837±0.0207	0.9788±0.0033	0.8792±0.0366	
MixedP	0.9761±0.0062	0.8663±0.0283	0.9827±0.0020	0.8723±0.0255	0.9799±0.0032	0.8830±0.0354	
Boldened numbers highlight the best performance within the same batch size. S represents the small angle set, L represents the large angle set, and the P represents our proposed angle set.

Table 2 Average (mean and standard deviation) of Dice coefficient score, for each of the rotation settings trained on 3D U-Net.

Rotation Setting	BS=1	BS=2	BS=4	
Train	Test	Train	Test	Train	Test	
Baseline	0.9135±0.1158	0.7689±0.1618	0.9362±0.0558	0.8622±0.0527	0.9391±0.0524	0.8487±0.0538	
AxialS	0.9312±0.0642	0.8330±0.0202	0.9629±0.0040	0.8703±0.0532	0.9579±0.0081	0.8435±0.0762	
AxialL	0.8886±0.1594	0.7546±0.1659	0.9235±0.0623	0.8291±0.1127	0.9578±0.0091	0.8471±0.0320	
AxialP	0.8998±0.0912	0.8164±0.0824	0.9570±0.0042	0.8652±0.0379	0.9575±0.0039	0.8590±0.0328	
CoronalL	0.9555±0.0060	0.8262±0.0268	0.9385±0.0480	0.8645±0.0809	0.9506±0.0042	0.8583±0.0411	
CoronalP	0.9027±0.0844	0.8059±0.0877	0.9647±0.0050	0.8841±0.0262	0.8842±0.1615	0.7903±0.1314	
SagittalS	0.8896±0.1649	0.8045±0.1400	0.9164±0.0067	0.8367±0.0934	0.9503±0.0088	0.8110±0.0954	
SagittalL	0.8944±0.1303	0.7819±0.1490	0.9097±0.0654	0.8274±0.0506	0.9061±0.0643	0.7862±0.0839	
MixedS	0.8762±0.0834	0.7949±0.0672	0.9307±0.0537	0.8279±0.0344	0.9579±0.0055	0.8677±0.0220	
MixedP	0.9086±0.0751	0.8156±0.1011	0.9589±0.0067	0.8818±0.0211	0.9266±0.0409	0.7954±0.0728	
Boldened numbers highlight the best performance within the same batch size. S represents the small angle set, L represents the large angle set, and the P represents our proposed angle set.

3D segmentation

In general, results from applying 3D rotations to 3D U-Net were mostly inconclusive.

Althought the best result from this section comes from the 3D coronal rotation with our proposed set of angles, the results in other batch sizes were too fluctuated for us to recommend this setting. Our proposed 3D axial rotation exhibits lower standard deviation across all batch sizes, compared to the baseline, improved segmentation accuracy on the test data is not significantly better in BS = 2. However, when combining this proposed 3D axial rotation setting with our proposed 3D coronal rotation, segmentation performance dropped except for BS = 2.

When considering the relative input size between 3D U-Net and U-Net, where BS = 1 (BS = 1 ×64×128×128) in 3D U-Net would be equivalent to BS = 64 (BS = 64 ×128×128) in U-Net, it seemed to exhibit that 3D U-Net produced inferior results compared to its 2D counterpart.Table 3 Dice efficient score for each rotation settings on U-Net model, across five validation folds.

Model	Batch	1st Fold	2nd Fold	3rd Fold	4th Fold	5th Fold	Average	
Name	Size	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	
Baseline	4	0.6840	0.5666	0.6867	0.4812	0.6845	0.5712	0.6859	0.5133	0.6851	0.4784	0.6852±0.0011	0.5221±0.0449	
8	0.7267	0.6260	0.7337	0.5297	0.7225	0.5950	0.7485	0.5545	0.7198	0.5666	0.7302±0.0115	0.5744±0.0372	
16	0.8482	0.7488	0.8731	0.6214	0.8517	0.7160	0.8624	0.6462	0.8711	0.6142	0.8613±0.0112	0.6693±0.0599	
32	0.9950	0.8798	0.9954	0.8608	0.9969	0.9042	0.9970	0.8553	0.9953	0.8782	0.9959±0.0010	0.8757±0.0192	
64	0.9907	0.8859	0.9897	0.8454	0.9949	0.9086	0.9968	0.8690	0.9939	0.8723	0.9932±0.0030	0.8762±0.0232	
128	0.9870	0.8951	0.9947	0.8146	0.9887	0.8990	0.9899	0.8738	0.9941	0.9032	0.9909±0.0034	0.8771±0.0367	
2D AxialL	4	0.6152	0.5190	0.6339	0.4580	0.6089	0.5389	0.6121	0.4758	0.6342	0.4904	0.6209±0.0122	0.4964±0.0326	
8	0.6209	0.5356	0.6269	0.4628	0.6188	0.5509	0.6160	0.5097	0.6310	0.4857	0.6227±0.0061	0.5090±0.0359	
16	0.6219	0.5230	0.6326	0.4710	0.6166	0.5347	0.6120	0.4877	0.6183	0.4951	0.6203±0.0078	0.5023±0.0261	
32	0.6078	0.5031	0.6363	0.4525	0.6176	0.5286	0.6200	0.5027	0.6219	0.4786	0.6207±0.0103	0.4931±0.0288	
64	0.6227	0.5283	0.6282	0.4640	0.6145	0.5479	0.6156	0.4840	0.6316	0.4907	0.6225±0.0075	0.5030±0.0342	
128	0.6205	0.5345	0.6316	0.4657	0.6125	0.5475	0.6110	0.5091	0.6215	0.4611	0.6194±0.0083	0.5036±0.0392	
AxialS	4	0.6478	0.5634	0.6513	0.4701	0.6368	0.5836	0.6600	0.5138	0.6777	0.4869	0.6547±0.0153	0.5236±0.0487	
8	0.7100	0.5956	0.7127	0.5536	0.7185	0.6318	0.7109	0.5531	0.7094	0.5496	0.7123±0.0037	0.5767±0.0361	
16	0.8201	0.7176	0.8654	0.6305	0.8378	0.7414	0.8357	0.6616	0.8500	0.6692	0.8418±0.0169	0.6841±0.0447	
32	0.9813	0.8786	0.9813	0.8738	0.9716	0.9063	0.9837	0.8418	0.9698	0.8828	0.9775±0.0064	0.8767±0.0231	
64	0.9786	0.8890	0.9743	0.8841	0.9824	0.9113	0.9723	0.8435	0.9786	0.8674	0.9772±0.0040	0.8791±0.0253	
128	0.9761	0.8988	0.9777	0.8650	0.9809	0.9104	0.9761	0.8830	0.9686	0.9113	0.9759±0.0045	0.8937±0.0197	
AxialL	4	0.6340	0.5445	0.6565	0.5100	0.6569	0.5452	0.6412	0.4911	0.6573	0.5029	0.6492±0.0109	0.5187±0.0248	
8	0.6847	0.5908	0.6967	0.5505	0.6969	0.6254	0.7078	0.5313	0.6972	0.5486	0.6967±0.0082	0.5693±0.0382	
16	0.7813	0.6982	0.8312	0.6217	0.8306	0.7416	0.8407	0.6740	0.8419	0.6524	0.8251±0.0251	0.6776±0.0455	
32	0.9660	0.8768	0.9675	0.8885	0.9671	0.9054	0.9705	0.8429	0.9640	0.8873	0.9670±0.0024	0.8802±0.0232	
64	0.9613	0.8704	0.9740	0.8786	0.9714	0.9183	0.9763	0.8752	0.9661	0.8542	0.9698±0.0061	0.8794±0.0237	
128	0.9677	0.9017	0.9762	0.8828	0.9658	0.9104	0.9665	0.8810	0.9564	0.8882	0.9665±0.0070	0.8928±0.0127	
AxialP	4	0.6606	0.5649	0.6543	0.4930	0.6550	0.5449	0.6428	0.4614	0.6570	0.4843	0.6540±0.0067	0.5097±0.0434	
8	0.6892	0.5914	0.6943	0.5450	0.7028	0.5967	0.7105	0.5480	0.7028	0.5411	0.6999±0.0083	0.5645±0.0272	
16	0.8315	0.7336	0.8435	0.6683	0.8316	0.7215	0.8460	0.6668	0.8626	0.6582	0.8430±0.0128	0.6897±0.0350	
32	0.9748	0.8756	0.9736	0.8761	0.9715	0.9112	0.9677	0.8576	0.9745	0.8786	0.9724±0.0029	0.8798±0.0194	
64	0.9733	0.8883	0.9680	0.8825	0.9722	0.9135	0.9694	0.8621	0.9775	0.8853	0.9721±0.0037	0.8863±0.0183	
128	0.9732	0.9064	0.9780	0.8662	0.9699	0.9097	0.9744	0.8861	0.9747	0.9147	0.9740±0.0029	0.8966±0.0202	
CoronalL	4	0.6296	0.5523	0.6974	0.4787	0.6552	0.5817	0.6631	0.4910	0.6465	0.4576	0.6584±0.0251	0.5123±0.0524	
8	0.6889	0.6006	0.7293	0.4944	0.7265	0.6098	0.7413	0.5536	0.7189	0.5682	0.7210±0.0197	0.5653±0.0458	
16	0.8462	0.7666	0.8563	0.6376	0.8416	0.7316	0.8515	0.6347	0.8589	0.6094	0.8509±0.0071	0.6760±0.0688	
32	0.9657	0.8747	0.9762	0.8705	0.9740	0.9135	0.9705	0.8028	0.9715	0.8738	0.9716±0.0040	0.8671±0.0400	
64	0.9748	0.8783	0.9714	0.8307	0.9787	0.9087	0.9765	0.8357	0.9764	0.8632	0.9756±0.0027	0.8633±0.0321	
128	0.9750	0.8978	0.9788	0.8424	0.9723	0.9030	0.9771	0.8605	0.9771	0.9160	0.9761±0.0025	0.8839±0.0311	
CoronalP	4	0.6466	0.5442	0.6469	0.4586	0.6478	0.5255	0.6523	0.5155	0.6586	0.5203	0.6504±0.0051	0.5128±0.0322	
8	0.7062	0.5740	0.7140	0.5089	0.7087	0.6081	0.7123	0.5180	0.6743	0.5509	0.7031±0.0164	0.5520±0.0408	
16	0.8495	0.7516	0.8829	0.6068	0.8388	0.7190	0.8242	0.6063	0.8649	0.6452	0.8521±0.0228	0.6658±0.0664	
32	0.9837	0.8844	0.9839	0.8482	0.9727	0.9041	0.9821	0.8310	0.9801	0.8838	0.9805±0.0046	0.8703±0.0298	
64	0.9870	0.8788	0.9785	0.8534	0.9728	0.9179	0.9794	0.8551	0.9787	0.8775	0.9793±0.0051	0.8765±0.0260	
128	0.9779	0.9016	0.9804	0.8204	0.9842	0.9022	0.9709	0.8605	0.9736	0.8943	0.9774±0.0053	0.8758±0.0354	
MixedS	4	0.6443	0.5264	0.6395	0.4960	0.6362	0.5781	0.6505	0.4570	0.6608	0.5522	0.6463±0.0097	0.5219±0.0474	
8	0.7030	0.6254	0.7180	0.5317	0.6667	0.5924	0.6989	0.4950	0.7065	0.6093	0.6986±0.0192	0.5708±0.0552	
16	0.8252	0.7497	0.8564	0.6211	0.8169	0.7525	0.8302	0.5830	0.8547	0.6766	0.8367±0.0179	0.6766±0.0757	
32	0.9742	0.8778	0.9734	0.8520	0.9811	0.9010	0.9818	0.8130	0.9876	0.8804	0.9796±0.0059	0.8648±0.0338	
64	0.9872	0.8763	0.9839	0.8663	0.9785	0.9147	0.9853	0.8669	0.9779	0.8945	0.9826±0.0042	0.8837±0.0207	
128	0.9738	0.9031	0.9817	0.8180	0.9818	0.8944	0.9784	0.8734	0.9781	0.9070	0.9788±0.0033	0.8792±0.0366	
MixedP	4	0.6326	0.5376	0.6638	0.5195	0.6343	0.5914	0.6418	0.4810	0.6631	0.4871	0.6471±0.0153	0.5233±0.0446	
8	0.7230	0.5883	0.7243	0.5425	0.7100	0.6393	0.7053	0.4675	0.7015	0.5986	0.7128±0.0103	0.5672±0.0655	
16	0.8307	0.7280	0.8640	0.5984	0.8272	0.7423	0.8166	0.5988	0.8462	0.6241	0.8369±0.0185	0.6583±0.0711	
32	0.9772	0.8790	0.9772	0.8423	0.9696	0.9031	0.9713	0.8337	0.9853	0.8734	0.9761±0.0062	0.8663±0.0283	
64	0.9805	0.8750	0.9851	0.8549	0.9809	0.9131	0.9832	0.8475	0.9838	0.8712	0.9827±0.0020	0.8723±0.0255	
128	0.9775	0.9060	0.9854	0.8277	0.9794	0.9054	0.9796	0.8673	0.9776	0.9088	0.9799±0.0032	0.8830±0.0354	
Averages (mean) are displayed in the last two columns. S represents the small angle set, L represents the large angle set, and the P represents our proposed angle set.

Table 4 Dice efficient score for each rotation settings on 3D U-Net model, across five validation folds.

Model	Batch	1st Fold	2nd Fold	3rd Fold	4th Fold	5th Fold	Average	
Name	Size	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	Train	Test	
Baseline	1	0.9661	0.8620	0.7064	0.4969	0.9609	0.8966	0.9702	0.8415	0.9640	0.7475	0.9135±0.1158	0.7689±0.1618	
2	0.9609	0.8915	0.9730	0.8454	0.9524	0.8829	0.8372	0.7786	0.9572	0.9128	0.9362±0.0558	0.8622±0.0527	
4	0.9717	0.8835	0.9614	0.8314	0.9555	0.8996	0.9609	0.8652	0.8459	0.7639	0.9391±0.0524	0.8487±0.0538	
AxialS	1	0.9711	0.8261	0.9551	0.8633	0.8170	0.8255	0.9543	0.8405	0.9583	0.8095	0.9312±0.0642	0.8330±0.0202	
2	0.9656	0.8651	0.9588	0.7837	0.9660	0.9127	0.9583	0.8754	0.9657	0.9147	0.9629±0.0040	0.8703±0.0532	
4	0.9635	0.8614	0.9599	0.7086	0.9612	0.8908	0.9610	0.8748	0.9436	0.8822	0.9579±0.0081	0.8435±0.0762	
AxialL	1	0.9662	0.7670	0.9623	0.8267	0.8454	0.6757	0.9651	0.8722	0.5975	0.4529	0.8886±0.1594	0.7546±0.1659	
2	0.9622	0.8859	0.8511	0.6303	0.9597	0.9010	0.9525	0.8783	0.8386	0.7980	0.9235±0.0623	0.8291±0.1127	
4	0.9664	0.8904	0.9671	0.8045	0.9496	0.8559	0.9517	0.8643	0.9494	0.8699	0.9578±0.0091	0.8471±0.0320	
AxialP	1	0.9687	0.8467	0.7725	0.7017	0.9578	0.8985	0.8330	0.7610	0.9668	0.8739	0.8998±0.0912	0.8164±0.0824	
2	0.9539	0.8840	0.9534	0.8295	0.9618	0.8900	0.9545	0.8195	0.9612	0.9029	0.9570±0.0042	0.8652±0.0379	
4	0.9586	0.8243	0.9585	0.8340	0.9533	0.8837	0.9540	0.8516	0.9629	0.9014	0.9575±0.0039	0.8590±0.0328	
CoronalL	1	0.9569	0.8246	0.9460	0.8476	0.9545	0.8931	0.9622	0.8712	0.9580	0.8765	0.9555±0.0060	0.8626±0.0268	
2	0.9679	0.9027	0.8533	0.7216	0.9633	0.9077	0.9552	0.8793	0.9529	0.9115	0.9385±0.0480	0.8645±0.0809	
4	0.9526	0.8878	0.9491	0.7885	0.9551	0.8876	0.9520	0.8708	0.9440	0.8565	0.9506±0.0042	0.8583±0.0411	
CoronalP	1	0.9667	0.8581	0.8477	0.7137	0.9566	0.9033	0.7805	0.7125	0.9620	0.8421	0.9027±0.0844	0.8059±0.0877	
2	0.9668	0.8737	0.9641	0.8482	0.9721	0.9086	0.9611	0.8790	0.9596	0.9110	0.9647±0.0050	0.8841±0.0262	
4	0.9607	0.8732	0.9620	0.8491	0.9430	0.7612	0.5955	0.5735	0.9595	0.8945	0.8842±0.1615	0.7903±0.1314	
SagittalS	1	0.9683	0.8174	0.9573	0.8890	0.9592	0.9035	0.5947	0.5613	0.9685	0.8511	0.8896±0.1649	0.8045±0.1400	
2	0.9659	0.9071	0.8502	0.6901	0.8345	0.7975	0.9685	0.8827	0.9629	0.9060	0.9164±0.0679	0.8367±0.0934	
4	0.9386	0.6973	0.9436	0.7240	0.9580	0.8958	0.9531	0.8373	0.9583	0.9006	0.9503±0.0088	0.8110±0.0954	
SagittalL	1	0.9442	0.8154	0.6617	0.5252	0.9548	0.8938	0.9654	0.8770	0.9458	0.7980	0.8944±0.1303	0.7819±0.1490	
2	0.9577	0.8296	0.9513	0.8057	0.9629	0.9113	0.8371	0.8130	0.8393	0.7774	0.9097±0.0654	0.8274±0.0506	
4	0.9595	0.8488	0.8454	0.6701	0.9500	0.8542	0.8269	0.7239	0.9487	0.8340	0.9061±0.0643	0.7862±0.0839	
MixedS	1	0.9624	0.8317	0.8494	0.7157	0.8220	0.8429	0.7822	0.7284	0.9649	0.8560	0.8762±0.0834	0.7949±0.0672	
2	0.8398	0.7830	0.9719	0.8310	0.9585	0.8661	0.9584	0.8547	0.9249	0.8048	0.9307±0.0537	0.8279±0.0344	
4	0.9556	0.8925	0.9609	0.8413	0.9597	0.8827	0.9636	0.8485	0.9495	0.8737	0.9579±0.0055	0.8677±0.0220	
MixedP	1	0.8329	0.6381	0.9599	0.8712	0.8200	0.8271	0.9636	0.8636	0.9665	0.8779	0.9086±0.0751	0.8156±0.1011	
2	0.9703	0.8860	0.9559	0.8644	0.9580	0.9128	0.9579	0.8597	0.9526	0.8859	0.9589±0.0067	0.8818±0.0211	
4	0.9526	0.8355	0.8574	0.6877	0.9233	0.7623	0.9410	0.8156	0.9585	0.8758	0.9266±0.0409	0.7954±0.0728	
Averages (mean) are displayed in the last two columns. S represents the small angle set, L represents the large angle set, and the P represents our proposed angle set.

Significant values are in bold.

Bold italic values indicate the best results.

Discussion

One clear conclusion from this experiment was that 2D slice-wise augmentation should be avoided, when segmenting 3D medical image volume. It also showed that our proposed 3D axial rotation maybe preferred over the baseline, when taking into account the segmentation accuracy over the test data and the standard deviation across five cross-validation folds.

For the significant performance drop when applying 2D rotation settings, we suspect that with relatively more variations between subsequent slices within the same batch may be more difficult for the underlying convolutional kernel in U-Net to account for all the variations. In contrast, when the rotation is more uniform across the same batch like in 3D rotation settings, it maybe easier for convolutional kernels to adapt for the more-consistent changes in rotation angles. This maybe the case especially when the batch size is low (i.e., U-Net with BS<8) in which the network has to update its weights during backpropagation more often and would result in more fluctuations in weight changes because of the larger variation in the alignment of input slices. Considering other related works like nnU-Net where large rotation angles were applied to 2D input slices—comparable to our 2D rotation settings—but with reduced augmentation frequency with the introduction of the probability of applying a rotation operation, applying large rotations less frequently may help compensate for the possible performance degradation caused by too much variability in input data exposed to the model during the training phase.

For 3D rotation settings, they seemed to help the model generalise better in unseen cases other than those used during training. This could be observed from higher accuracy on the test data, though the accuracy on the training data might be lower compared to the baseline. This is because the model is exposed to a wider variability of data during training, to both the original data and the augmented data, and since these augmentations were done on-the-fly so this would reflect in reduced training accuracy. However, as the model learned to generalise to more training samples rather than just the original unmodified data, it showed improved performance in the test set, which was not included during the training phase.

While results from applying 3D coronal rotations were inferior than 3D axial rotations, there might be benefit yet to be realised from rotating in other axes especially when segmenting with other axes than the axial axis. This would be the case for multi-axes segmentation models that incorporate segmented information from different axes to form judgements on the final segmentation.

It could also be seen in Fig. 7 that the 2D rotation setting produced more artefacts outside the cardiac structure, that the model incorrectly predicted pixels outside the region of interest as its result. Other segmented results are otherwise relatively similar, as observed from Fig. 8, with varying segmentation accuracy around the boundary of cardiac chambers. To address this issue, techniques like localising the region of interest and masking background pixels before segmenting for cardiac chambers might be able to help improve the performance, as demonstrated by Xia et al.3. This technique comes with an additional computation cost associated with introducing an extra localisation step, and with low false positive artefacts produced from the baseline model we suspect that lessen the augmentation frequency as described previously would be a more effective solution.

In case where the performance of segmenting with the 3D U-Net model are inferior to those of the U-Net, we suspect that it might be due to non-optimal settings for model hyperparameters. 3D segmentation models, or models with more learnable parameters in general, would normally be expected to have higher sensitivity to changes in model hyperparameters. With 3D U-Net’s approximately 19 million learnable parameters compared to approximately 7 million in U-Net, further fune-tuning may help improve 3D U-Net’s performance.

Finally, it maybe more fruitful to experiment 2D and 3D rotation settings in other datasets with more datapoints, to provide further insights on how 2D and 3D rotation differs in different medical image domain. However, in reference to our proposed rotation according to probably anatomical alignment of cardiac substructures, different cardiac datasets or datasets on other organs than the heart may require further analyses by medical experts to provide a candidate range of rotation angles suitable for the organ and the dataset in question.Figure 7 2D results from the middle slice of a sample volume. The first column on the first row shows the annotated ground truth, with each colour denotes each cardiac chambers. For other images, green denotes true positive results whereas red denotes both false positive and false negative results.

Figure 8 3D results of a sample volume, from a selection of modelsand rotation configurations (BS = 4). Left atrium (LA) is coloured in red, right atrium (RA) in yellow, left ventricle (LV) in green, and right ventricle (RV) in blue respectively.

Acknowledgements

This research/project was undertaken with the assistance of resources and services from the National Computational Infrastructure (NCI), which is supported by the Australian Government.

Author contributions

W.K. and K.S. conceived the experiments. K.S. conducted the experiments. T.A. contributed medical opinions to both the experiment settings and the result evaluation. W.K. and M.O. analysed the results and refine the experiments. All authors reviewed the manuscript.

Data availability

Access to the Multi-dimensionality Whole Heart Segment (MM-WHS) dataset can be requested on this link https://zmiclab.github.io/zxh/0/mmwhs.

Competing interests

The authors declare no competing interests.

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

These authors contributed equally: Krittanat Sutassananon, Worapan Kusakunniran, Mehmet Orgun and Thanongchai Siriapisith
==== Refs
References

1. Zhuang X Evaluation of algorithms for multi-modality whole heart segmentation: An open-access grand challenge Med. Image Anal. 2019 58 101537 10.1016/j.media.2019.101537 31446280
Zhuang, X. et al. Evaluation of algorithms for multi-modality whole heart segmentation: An open-access grand challenge. Med. Image Anal. 58, 101537. 10.1016/j.media.2019.101537 (2019).31446280 10.1016/j.media.2019.101537
2. Xiong Z A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging Med. Image Anal. 2021 67 101832 10.1016/j.media.2020.101832 33166776
Xiong, Z. et al. A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging. Med. Image Anal. 67, 101832. 10.1016/j.media.2020.101832 (2021).33166776 10.1016/j.media.2020.101832
3. Xia, Q., Yao, Y., Hu, Z. & Hao, A. Automatic 3D atrial segmentation from GE-MRIs using volumetric fully convolutional networks. In Statistical Atlases and Computational Models of the Heart. Atrial Segmentation and LV Quantification Challenges. Vol. 11395. 211–220.10.1007/978-3-030-12029-0_23 (Springer, 2019).
4. Ronneberger, O., Fischer, P. & Brox, T. U-Net: Convolutional Networks for Biomedical Image Segmentation (2015).
5. Çiçek, O., Abdulkadir, A., Lienkamp, S. S., Brox, T. & Ronneberger, O. 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation (2016).
6. Milletari, F., Navab, N. & Ahmadi, S.-A. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation (2016).
7. Gu Z CE-Net: Context encoder network for 2D medical image segmentation IEEE Trans. Med. Imaging 2019 38 2281 2292 10.1109/TMI.2019.2903562 30843824
Gu, Z. et al. CE-Net: Context encoder network for 2D medical image segmentation. IEEE Trans. Med. Imaging 38, 2281–2292. 10.1109/TMI.2019.2903562 (2019).30843824 10.1109/TMI.2019.2903562
8. Baldeon-Calisto M Lai-Yuen SK AdaResU-Net: Multiobjective adaptive convolutional neural network for medical image segmentation Neurocomputing 2020 392 325 340 10.1016/j.neucom.2019.01.110
Baldeon-Calisto, M. & Lai-Yuen, S. K. AdaResU-Net: Multiobjective adaptive convolutional neural network for medical image segmentation. Neurocomputing 392, 325–340. 10.1016/j.neucom.2019.01.110 (2020).10.1016/j.neucom.2019.01.110
9. Xu, Z., Wu, Z. & Feng, J. CFUN: Combining Faster R-CNN and U-net Network for Efficient Whole Heart Segmentation (2018).
10. Chen, J. et al. TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation (2021).
11. Isensee F Jaeger PF Kohl SAA Petersen J Maier-Hein KH nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation Nat. Methods 2021 18 203 211 10.1038/s41592-020-01008-z 33288961
Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J. & Maier-Hein, K. H. nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18, 203–211. 10.1038/s41592-020-01008-z (2021).33288961 10.1038/s41592-020-01008-z
12. Oktay, O. et al. Attention U-Net: Learning Where to Look for the Pancreas (2018).
13. Ibtehaz, N. & Rahman, M. S. MultiResUNet : Rethinking the U-Net architecture for multimodal biomedical image segmentation. Neural Netw. 121, 74–87. 10.1016/j.neunet.2019.08.025 (2020).
14. Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization (2017).
