
==== Front
Phys Imaging Radiat Oncol
Phys Imaging Radiat Oncol
Physics and Imaging in Radiation Oncology
2405-6316
Elsevier

S2405-6316(24)00110-6
10.1016/j.phro.2024.100640
100640
Original Research Article
Artificial intelligence-generated targets and inter-observer variation in online adaptive radiotherapy of bladder cancer
Åström Lina M. lina.frida.moeller.aastroem@regionh.dk
ab⁎
Sibolt Patrik a
Chamberlin Hannah c
Serup-Hansen Eva a
Andersen Claus E. b
van Herk Marcel c
Mouritsen Lene S. a
Aznar Marianne C. c
Behrens Claus P. ab
a Department of Oncology, Copenhagen University Hospital – Herlev and Gentofte, Copenhagen, Denmark
b Department of Health Technology, Technical University of Denmark, Roskilde, Denmark
c Division of Cancer Sciences, School of Medical Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester, United Kingdom
⁎ Corresponding author at: Copenhagen University Hospital – Herlev and Gentofte, Radiotherapy Research Unit (52AA), Department of Oncology, Borgmester Ib Juuls vej 7, DK-2730 Herlev, Denmark. lina.frida.moeller.aastroem@regionh.dk
01 9 2024
7 2024
01 9 2024
31 10064024 5 2024
29 8 2024
29 8 2024
© 2024 The Author(s)
2024
https://creativecommons.org/licenses/by/4.0/ This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
Highlights

• Small inter-observer variation in targets manually edited by radiotherapy technicians.

• Delineation uncertainty was non-isotropic.

• Artificial intelligence algorithms consistently underestimated the bladder target.

• Manual adjustment of artificial intelligent generated contours was necessary to ensure target coverage.

Background and purpose

Daily target re-delineation in online adaptive radiotherapy (oART) introduces uncertainty. The aim of this study was to evaluate artificial intelligence (AI) generated contours and inter-observer target variation among radiotherapy technicians in cone-beam CT (CBCT) guided oART of bladder cancer.

Materials and methods

For each of 10 consecutive patients treated with oART for bladder cancer, one CBCT was randomly selected and retrospectively included. The bladder (CTV-T) was AI-segmented (CTV-TAI). Seven radiotherapy technicians independently reviewed and edited CTV-TAI, generating CTV-TADP. Contours were benchmarked against a ground truth contour (CTV-TGT) delineated blindly from scratch. CTV-TADP and CTV-TAI were compared to CTV-TGT using volume, dice similarity coefficient, and bidirectional local distance. Dose coverage (D99%>95 %) of CTV-TGT was evaluated for treatment plans optimized for CTV-TAI and CTV-TADP with clinical margins. Inter-observer variation among CTV-TADP was assessed using coefficient of variation and generalized conformity index.

Results

CTV-TGT ranged from 48.7 cm3 to 211.6 cm3. The median [range] volume difference was 4.5 [−17.8, 42.4] cm3 for CTV-TADP and −15.5 [−54.2, 4.3] cm3 for CTV-TAI, compared to CTV-TGT. Corresponding dice similarity coefficients were 0.87 [0.71, 0.95] and 0.84 [0.64, 0.95]. CTV-TGT was adequately covered in 68/70 plans optimized on CTV-TADP and in 6/10 plans optimized on CTV-TAI with clinical margins. The median [range] coefficient of variation was 0.08 [0.05, 0.11] and generalized conformity index was 0.78 [0.71, 0.88] among CTV-TADP.

Conclusions

Target re-delineation in CBCT-guided oART of bladder cancer demonstrated non-isotropic inter-observer variation. Manual adjustment of AI-generated contours was necessary to cover ground truth targets.
==== Body
pmc1 Introduction

Target delineation variation is a major challenge in radiotherapy. Together with anatomical variations, it is one of the largest uncertainties to consider during treatment planning [1], [2]. Traditionally, in the margin concept first introduced by van Herk et al., the delineation uncertainty is defined as a systematic component that leads to a displacement of the dose distribution with respect to the clinical target volume (CTV) [3]. While this is true for conventional non-adaptive image-guided radiotherapy, where targets are delineated once prior to treatment start, it may become more of a random uncertainty in online adaptive radiotherapy (oART), where targets are re-delineated daily, often by different observers. However, it may still have a systematic component, e.g., due to influence or guidance by reference contours in the re-delineation procedure.

The use of oART has vastly increased during the last decade, through the introduction of advanced systems, some of which utilizing artificial intelligence (AI) [4], [5]. Several studies have demonstrated the clinical feasibility of cone-beam computed tomography (CBCT) guided oART for various disease sites, especially in the pelvis [6], [7], [8], [9], [10], [11]. When the treatment is completely re-optimized to the daily anatomy, any inter-fractional anatomical variations are considered, enabling the margins to be reduced. For bladder cancer, we have previously reported large margin reductions, resulting in sparing of organs at risk (OARs) [12], [13]. However, if not accounted for, uncertainties related to AI-generated contours and daily re-delineation of targets may have a negative effect on the clinical outcome.

The magnitude of the delineation uncertainty depends on several factors, such as image quality (modality, resolution, contrast, artifacts), observer (experience and educational background), use of delineation guidelines, format of contouring (from scratch, automatic segmentation, or propagation from a reference image), and time pressure (workload/throughput) [1], [14], [15], [16], [17], [18], [19]. Previous studies have investigated inter-observer CTV delineation variation for bladder cancer on CBCT [20], [21], but the contours were delineated by physicians without any time constraint. Since the anatomy within the pelvis is highly dynamic, time-efficient delineation is crucial in oART. Furthermore, to reduce the need for daily online support by physician and thus make oART clinically practicable, radiotherapy technicians are increasingly taking on responsibilities previously attributed to the physicians [22], [23]. The aim of this study was to assess inter-observer CTV variation in oART of bladder cancer, among radiotherapy technicians performing oART routinely. Additionally, the quality of AI-generated bladder contours and the need for editing them were evaluated. Contours were benchmarked against ground truth contours to assess the impact of identified discrepancies on dose coverage.

2 Materials and methods

2.1 Patients and observers

This retrospective study included 10 consecutive patients treated with curative-intended CBCT-guided oART for muscle-invasive bladder cancer between February 2020 and February 2021 (Supplementary Table S1). Patient cases included three males and seven females, one patient had a hip prosthesis, and three patients had a catheter. A total dose of 64 Gy in 32 fractions was prescribed to the primary planning target volume (PTV-T), according to Danish national and international guidelines [24], [25]. All patients were treated at an Ethos Therapy v1.1 treatment unit (Varian, a Siemens Healthineers Company). Further details on the treatment are included in a previous publication [12].

A total of seven adapters participated in this study. The adapters consisted of radiotherapy technicians (4 nurses and 3 radiographers) that had completed a one-year national training program in radiotherapy as well as a departmental training program in CBCT-guided oART and who routinely carried out oART clinically without daily support by physicians and/or physicists. The training program included anatomy and oART lectures by physicians and physicists, supervised and non-supervised offline delineations, and online supervised delineations. This study was approved by the department as part of the quality assurance program for CBCT-guided oART. All patients were anonymized during the conduction and analysis of the study.

2.2 Contouring guidelines and margins

Reference CT and magnetic resonance imaging (MRI) scans were acquired approximately one week prior to treatment start (scanning details in Supplementary Materials). The patients were instructed to have an empty bladder and rectum during reference scans and treatments. They were instructed to avoid drinking fluids two hours prior to both scans and treatments to reduce the intra-fractional anatomical variation.

The primary CTV (CTV-T) was defined as the outer wall of the bladder including any visible tumour as seen on CT, with possible extra-bladder extension caudally as guided by the reference MRI. When indicated according to Danish national guidelines [24], the CTV-T involved prostate and/or seminal vesicles.

PTV-T was generated by adding patient-specific anisotropic margins of 5–12 mm to CTV-T (Supplementary Table S1). Generally, the margins were 5 mm plus the maximum intra-fractional bladder variation as measured on pre- and post-adaptation CBCTs during two to four non-adaptive fractions prior to the start of oART. The intra-fractional bladder variation was measured in six directions (cranial, caudal, anterior, posterior, left, right) using anatomical landmarks (e.g., symphysis, femoral heads, and spinal column) as reference measuring points [12].

2.3 Inter-observer delineation variation

One session CBCT was randomly selected among the oART sessions in each of the 10 patients’ treatment courses and imported to an emulator with Ethos Treatment Management System v1.1. The CBCTs were acquired on Ethos Therapy v1.1 with the settings reported in Supplementary Table S1 and reconstructed with a slice thickness of 2 mm and a pixel size of 0.96 mm × 0.96 mm. Each adapter independently and blindly simulated one selected oART session per patient. In the oART session, the bladder was auto-contoured from scratch by integrated AI-algorithms, which followed the version of the system and did not learn based on input from the user (neither reference delineations nor previous fractions). The adapters reviewed and manually edited the AI-generated bladder contour, generating CTV-TADP, within a clinical time frame, i.e., as if it was a clinical treatment (typically within 10 min [12]). Just as in the clinical workflow, the reference CT and -contours was available during the simulations [12]. An additional oART session was simulated for each patient without any edit of the AI-generated contours to generate CTV-TAI. In parallel, a ground truth CTV-T (CTV-TGT) was first delineated by a senior oncologist, then reviewed by an oART experienced medical physicist, and finally edited to a consensus contour, all blind to the adapter’s delineations but with reference CT and -delineations available. Contouring of CTV-TGT was carried out in Eclipse treatment planning system v15.1 (Varian, a Siemens Healthineers Company) without any time constraint. If the prostate and seminal vesicles were included in the reference CTV-T, only the bladder part was included in CTV-TADP, CTV-TAI, and CTV-TGT. Further details on Ethos and the oART workflow can be found elsewhere [4], [12].

2.4 Data analysis

All CTV-TADP and CTV-TAI were compared to corresponding CTV-TGT using volume, dice similarity coefficient (DSC), and bidirectional local distance (BLD). DSC was calculated in Eclipse and BLD was calculated using Python. BLD was calculated at each point on the surface of CTV-TGT (pGT), by calculating: 1) the minimum absolute distance from pGT to CTV-TADP or CTV-TAI, and 2) the minimum absolute distances from each point on the comparator surface to CTV-TGT and selecting the largest of distances connected to pGT. BLD at pGT was then defined as the maximum of 1) and 2) [26].

The dose covering 99 % of the CTV-TGT and PTV-TGT (D99%>60.8 Gy in accordance with departmental guidelines) was evaluated for treatment plans optimized on PTV-TADP as well as PTV-TAI, i.e., CTV-TADP and CTV-TAI plus the clinical patient-specific CTV-T-to-PTV-T margins in Supplementary Table S1. PTV-TGT was evaluated as coverage of a larger volume than CTV-TGT was desired due to the large intra-fractional variations of the bladder [27], [28]. However, since PTV-TGT accounted for several uncertainties and the dose coverage of it is sensitive to small differences between CTV-TGT and CTV-TADP or CTV-TAI, complete coverage was not necessary nor expected. All plans were generated in Ethos as IMRT with 12 equidistant fields.

The inter-observer delineation variation was evaluated by calculating the generalized conformity index (CIgen) and coefficient of variation (CV) among CTV-TADP, for each patient. CIgen was defined as the ratio of the sum of all overlapping volumes between pairs of observers and the total volume of the same pair, i.e., CIgen=∑pairs ij|Vi ∩ Vj| / ∑pairs ij|Vi ∪ Vj| [29]. CV was calculated as the ratio between the standard deviation and the mean volume.

3 Results

Fig. 1 presents the nine contours (one CTV-TGT, one CTV-TAI, and seven CTV-TADP) on two different patients: one with relatively low and another with relatively high inter-observer delineation variation. Over all 10 patients, the CTV-TGT ranged from 48.7 cm3 to 211.6 cm3. The median [range] volume difference was 4.5 [−17.8, 42.4] cm3 for CTV-TADP and −15.5 [−54.2, 4.3] cm3 for CTV-TAI (Fig. 2, Supplementary Table S1). This corresponds to relative mean [range] volume differences of 5.3 % [−18.6 %, 36.5 %] and −18.8 % [−51.7 %, 6.1 %], respectively. CTV-TAI was smaller than all corresponding CTV-TGT except one and all corresponding CTV-TADP. The results for DSC and BLD comparing CTV-TADP and CTV-TAI to CTV-TGT are shown in Fig. 3. The median [range] DSC was 0.87 [0.71, 0.95] for CTV-TADP and 0.84 [0.64, 0.95] for CTV-TAI. Two-dimensional colour maps of the mean absolute BLD across all adapters at each point on CTV-TGT (Fig. 4), demonstrated that the areas with largest deviation of CTV-TADP compared to CTV-TGT were patient-dependent but primarily in cranial, caudal, anterior, and posterior directions. Studying the deviations for each CTV-TADP (Supplementary Figure S1), the deviations were not isotropic but concentrated to specific areas that varied among the patients. The BLD comparing CTV-TAI and CTV-TGT, varied over the volume and among the patients, but the largest absolute BLD was observed in cranial, caudal, and anterior direction (2-D maps are included in Supplementary Figure S2 and S3).Fig. 1 Sagittal view of CBCT with contours of CTV-TGT (yellow), CTV-TAI (red), CTV-TADP (remaining colours) for patient 1 (left) and 10 (right), with relatively low and high inter-observer delineation variation, respectively. CIgen was 0.88 for patient 1 and 0.71 for patient 10. Both patients have given written consent to use of images and treatment plans. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 2 Identity plot with volume of CTV-TADP (black dots) and CTV-TAI (red triangles) as a function of CTV-TGT volume. From the left, the patients are in the following order: 3, 5, 2, 4, 6, 9, 7, 10, 8, and 1. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 3 (Top) Dice Similarity Coefficient (DSC), (middle) mean absolute Bilateral Local Distance (BLD), and (bottom) maximum absolute BLD comparing CTV-TADP (black dots) and CTV-TAI (red triangles) with CTV-TGT for each patient. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 4 Viewed from the right side of the patient, i.e., anterior is to the right and posterior is to the left in each Fig. 2-D colour maps of mean of the absolute bidirectional local distance (BLD) across all seven adapters for patient 1–10, comparing CTV-TGT and CTV-TADP. The patients are ordered in a left–right, top–bottom manner. The colour scale goes from 0 mm to 5 mm, where values ≥ 5 mm are coloured in red. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Treatment plans optimized on PTV-TADP and PTV-TAI did not cover (D99%<60.8 Gy) CTV-TGT in 2/70 (patient 10) and 4/10 (patient 2, 5, 8, and 9) plans, respectively (Fig. 5). Dose coverage of CTV-TGT was lacking in anterior and cranial directions for PTV-TADP plans and in cranial, caudal, and anterior directions for PTV-TAI plans. PTV-TGT was not covered in 45/70 and 10/10 plans, respectively (Fig. 5).Fig. 5 D99% of CTV-TGT (left) and PTV-TGT (right) for plans optimized on CTV-TADP (black dots) and CTV-TAI (red triangles) with clinical CTV-T-to-PTV-T margins. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Among CTV-TADP, the median [range] CIgen was 0.78 [0.71, 0.88] and CV was 0.08 [0.05, 0.11]. Colour maps of two standard deviations of absolute BLD among CTV-TADP show that the largest disagreement among the adapters was in the cranial, anterior, and posterior directions (2D maps are included in Supplementary Figure S4).

4 Discussion

We present novel data on inter-observer target variation among adapters and quality of AI-generated contours in CBCT-guided oART of bladder cancer. Our results on CIgen and CV among CTV-TADP are comparable to other studies on inter-observer target variation of bladder cancer among physicians on CBCT. Foroudi et al. [20] report a mean CI (defined as the ratio between overlapping volumes and the total volume) of 0.75 for four patients and four radiation oncologists, and Nishioka et al. [21] report a mean CIgen of 0.81 and a mean CV of 0.08 for ten patients and five radiation oncologists. Studying the colour maps in Supplementary Figure S4, the areas with largest disagreement among CTV-TADP were primarily in the cranial, anterior, and posterior directions. This might be due to motion artefacts from bowel movements and similarity in Hounsfield unit values that challenges the distinguishment of bladder from surrounding organs.

In contrast to the studies by Foroudi et al. [20] and Nishioka et al. [21], where the contours were delineated from scratch, the observer’s contours (CTV-TADP) in this study were adapted from AI-generated contours. This may influence the contours and the related uncertainty. Similarly, this constitutes a possible bias compared to CTV-TGT, which was delineated from scratch without having the AI-contour as a start. However, it reflects the clinical situation at our as well as other institutions, where the daily CTV-T is based on an AI-generated bladder contour with manual edits [12]. Unlike the oART workflow described by Archambault et al. [4], our daily CTV-T is set as a copy of the (edited) AI-generated bladder contour to avoid unwanted deformations during the integrated automatic target propagation procedure. Other potential biases in CTV-TADP include awareness of participation in the study, which may affect the adapters’ delineation although they were instructed to delineate as in a clinical setting.

Our analysis revealed that AI consistently underestimated the CTV-T (Fig. 2). For patient 8, who had irregular bladder shape with small air cavities in cranial parts (probably due to tumour burden), the AI underperformed more than for other patients (Fig. 3). The evaluation of dose coverage showed that CTV-TAI was generally not acceptable without editing (Fig. 5). Discrepancies between CTV-TAI and CTV-TGT were expected because CTV-TAI was entirely CBCT-based while CTV-TGT was not necessarily so. The reference CTV-T on the reference CT, used during delineation of CTV-TGT and CTV-TADP, was guided by a reference MRI in the caudal parts for all patients except patient 4 and 6. These discrepancies could hypothetically be accounted for by a population-based margin, potentially reducing contour editing time and making the oART workflow more time-efficient. However, not only the areas of largest deviation but also the size of the deviations in CTV-TAI compared to CTV-TGT varied among the patients (Supplementary Figure S2). Dose coverage was lacking in caudal areas of CTV-TGT in plans optimized on PTV-TAI for some patients, though not for most of them. Therefore, our data does not support concluding on a population-based margin that would be valid for the majority of patients. However, extra attention should be paid to caudal, cranial, and anterior areas of the CTV-TAI. The need to edit the AI-generated contours in CBCT-guided oART has been recognized in other studies. Studies reporting on bladder cancer however is lacking; a study by Azzarouali et al. [7] primarily report on the deformation of boosted target volume rather than the AI generated contours for oART of bladder cancer with integrated boost. Byrne et al. [6] report that edit of AI-generated bladder and rectum contours was done in 89 % of the sessions for prostate cancer and Bak et al. [11] report edit of AI-generated bladder contour in 67 % of the sessions for vulvar cancer. While both report that the level of edit usually was small, the fact that the bladder was an OAR and not a target in both studies may affect the perceived level of adjustment needed. Furthermore, as the accuracy of the AI-generated contours depend on the system’s version, improvements in future versions might affect the need for edits.

The patients for whom CTV-TGT was under-covered differed for plans optimized on PTV-TAI and PTV-TADP. The two plans optimized on PTV-TADP that did not cover CTV-TGT both belonged to patient 10 (Fig. 5). This might be explained by the lower image quality observed for this patient. By selecting a random CBCT among clinical oART fractions for this study, CBCT of good enough quality for oART was assumed. However, this was not the case for patient 10, who only received 6 oART fractions, after which it was decided to change to a non-adaptive workflow due to difficulty in distinguishing bladder from bowel in the cranial direction. It could be argued that the CBCT quality for this patient is not representative of a typical oART situation, and this could possibly warrant excluding this patient from the analysis. However, we believe it reflects the reality of implementing and performing oART clinically and thus decided not to exclude the data from the analysis. Nevertheless, this reveals a limitation of this study: the absence of image quality scoring. While we have included data on CBCT acquisition and reconstruction parameters, a scoring of image quality, as in the study by Foroudi et al. [20], or delineation confidence could contribute important information.

For many patients and adapters, CTV-TADP was larger than CTV-TGT (Fig. 2). There was not a general over-delineation over the entire CTV-T, but rather deviations concentrated to specific areas that were patient-specific (Supplementary Figure S1). For some patients, there was a consistency over all adapters, where all CTV-TADP deviated from CTV-TGT in the same area, e.g., caudally for patient 2. The consistent discrepancy caudally for patient 2 may be explained by the difficulty in distinguishing the caudal bladder border on CBCT and level of experience of the observer. It highlights the challenge with having an MR-guided delineation in CBCT-guided oART workflow and what areas to focus on in the further education of the adapters. Furthermore, it highlights possible limitations of having a single “ground truth” contour, where CTV-TGT itself includes an uncertainty.

Unlike the independent and blinded delineations of CTV-TADP in this study, the adapters always work in pairs during the clinical treatments so that the daily contours are reviewed daily by a second adapter. We, furthermore, have a physician and a physicist present the first fraction, and weekly independent offline checks of contours and treatment plans by physicians and physicists. These efforts aim to reduce the inter-observer delineation variation. Multidisciplinary attendances during the first fractions provides the opportunity to discuss areas that need extra attention, either due to tumour burden and delineation difficulty, attempting to ensure the quality of contours and dose coverage of the disease. This study supports the need for such multidisciplinary efforts and continued feedback loops.

With the future aim to decide on a margin contribution of delineation uncertainty in oART, this study present novel data showing that adapters successfully re-delineate targets, supporting them as the main driver of the oART workflow. It also highlights the essential role of image quality [30]. With recent technological developments, the quality of CBCT images has significantly been improved [31]. Faster acquisition and improved reconstruction algorithms have demonstrated an image quality comparable with fan-beam CT [32]. This has the potential to reduce the delineation variation further.

In conclusion, target re-delineation in daily CBCT-guided oART of bladder cancer demonstrated non-isotropic inter-observer variation among adapters, suggesting non-isotropic margins to account for it. Manual adjustment of AI-generated contours resulted in improved accuracy in target delineation compared to ground truth and was necessary to ensure target coverage for some patients.

CRediT authorship contribution statement

Lina M. Åström: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Validation, Visualization, Writing – original draft. Patrik Sibolt: Conceptualization, Methodology, Project administration, Supervision, Validation, Writing – review & editing. Hannah Chamberlin: Methodology, Software, Visualization, Writing – review & editing. Eva Serup-Hansen: Conceptualization, Methodology, Supervision, Writing – review & editing. Claus E. Andersen: Conceptualization, Methodology, Supervision, Writing – review & editing. Marcel van Herk: Conceptualization, Investigation, Writing – review & editing. Lene S. Mouritsen: Investigation, Resources, Writing – review & editing. Marianne C. Aznar: Conceptualization, Methodology, Writing – review & editing. Claus P. Behrens: Conceptualization, Methodology, Project administration, Supervision, Validation, Writing – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A Supplementary data

The following are the Supplementary data to this article:Supplementary Data 1

Acknowledgements

We thank all participating adapters for contributing with essential delineations. MA acknowledges the support of the 10.13039/100000084 Engineering and Physical Research Council (Grant number EP/T028017/1 ), 10.13039/501100000289 Cancer Research UK RadNet Manchester (C1994/A28701 ) and the 10.13039/100014653 NIHR Manchester Biomedical Research Centre (NIHR203308 ). MVH is supported by the 10.13039/100014653 NIHR Manchester Biomedical Research Centre (NIHR203308 ) and this work was supported by the 10.13039/501100017008 Cancer Research UK Manchester Centre award (CTRQQR-2021\ 100010 ).

Appendix A Supplementary data to this article can be found online at https://doi.org/10.1016/j.phro.2024.100640.
==== Refs
References

1 Njeh C. Tumor delineation: The weakest link in the search for accuracy in radiotherapy J Med Phys 33 2008 136 140 10.4103/0971-6203.44472 19893706
2 Meijer G.J. Rasch C. Remeijer P. Lebesque J.V. Three-dimensional analysis of delineation errors, setup errors, and organ motion during radiotherapy of bladder cancer Int J Radiat Oncol Biol Phys 55 2003 1277 1287 10.1016/S0360-3016(02)04162-7 12654438
3 Van Herk M. Remeijer P. Rasch C. Lebesque J.V. The probability of correct target dosage: Dose-population histograms for deriving treatment margins in radiotherapy Int J Radiat Oncol Biol Phys 47 2000 1121 1135 10.1016/S0360-3016(00)00518-6 10863086
4 Archambault Y. Boylan C. Bullock D. Morgas T. Peltola J. Ruokokoski E. Making on-line adaptive radiotherapy possible using artificial intelligence and machine learning for efficient daily re-planning Med Phys Int J 8 2020 77 86
5 Winkel D. Bol G.H. Kroon P.S. van Asselen B. Hackett S.S. Werensteijn-Honingh A.M. Adaptive radiotherapy: The Elekta Unity MR-linac concept Clin Transl Radiat Oncol 18 2019 54 59 10.1016/j.ctro.2019.04.001 31341976
6 Byrne M. Archibald-Heeren B. Hu Y. Teh A. Beserminji R. Cai E. Varian ethos online adaptive radiotherapy for prostate cancer: Early results of contouring accuracy, treatment plan quality, and treatment time J Appl Clin Med Phys 23 2022 10.1002/acm2.13479 e13479
7 Azzarouali S. Goudschaal K. Visser J. Hulshof M. Admiraal M. van Wieringen N. Online adaptive radiotherapy for bladder cancer using a simultaneous integrated boost and fiducial markers Radiat Oncol 18 2023 165 10.1186/s13014-023-02348-8 37803392
8 de Jong R. Visser J. van Wieringen N. Wiersma J. Geijsen D. Bel A. Feasibility of Conebeam CT-based online adaptive radiotherapy for neoadjuvant treatment of rectal cancer Radiat Oncol 16 2021 136 10.1186/s13014-021-01866-7 34301300
9 Shelley C.E. Bolt M.A. Hollingdale R. Chadwick S.J. Barnard A.P. Rashid M. Implementing cone-beam computed tomography-guided online adaptive radiotherapy in cervical cancer Clin Transl Radiat Oncol 40 2023 100596 10.1016/j.ctro.2023.100596
10 Åström L.M. Behrens C.P. Storm K.S. Sibolt P. Serup-Hansen E. Online adaptive radiotherapy of anal cancer: Normal tissue sparing, target propagation methods, and first clinical experience Radiother Oncol 176 2022 92 98 10.1016/j.radonc.2022.09.015 36174846
11 Bak M.E. Jensen N.K.G. Nøttrup T.J. Mathiesen H.F. Roed H. Sjölin M. Clinical experiences with online adaptive radiotherapy of vulvar carcinoma Acta Oncol 62 2023 1230 1238 10.1080/0284186X.2023.2257377 37713179
12 Åström L.M. Behrens C.P. Calmels L. Sjöström D. Geertsen P. Mouritsen L.S. Online adaptive radiotherapy of urinary bladder cancer with full re-optimization to the anatomy of the day: Initial experience and dosimetric benefits Radiother Oncol 171 2022 37 42 10.1016/j.radonc.2022.03.014 35358605
13 Sibolt P. Andersson L.M. Calmels L. Sjöström D. Bjelkengren U. Geertsen P. Clinical implementation of artificial intelligence-driven cone-beam computed tomography-guided online adaptive radiotherapy in the pelvic region Phys Imaging Radiat Oncol 17 2021 1 7 10.1016/j.phro.2020.12.004 33898770
14 Simões R. Wortel G. Wiersma T.G. Janssen T.M. van der Heide U.A. Remeijer P. Geometrical and dosimetric evaluation of breast target volume auto-contouring Phys Imaging Radiat Oncol 12 2019 38 43 10.1016/j.phro.2019.11.003 33458293
15 Rasch C. Barillot I. Remeijer P. Touw A. van Herk M. Lebesque J.V. Definition of the prostate in CT and MRI: a multi-observer study Int J Radiat Oncol Biol Phys 43 1999 57 66 10.1016/S0360-3016(98)00351-4 9989514
16 Ng S.P. Cardenas C.E. Elhalawani H. Pollard C. Elgohari B. Fang P. Comparison of tumor delineation using dual energy computed tomography versus magnetic resonance imaging in head and neck cancer re-irradiation cases Phys Imaging Radiat Oncol 14 2020 1 5 10.1016/j.phro.2020.04.001 33458306
17 Weiss E. Richter S. Krauss T. Metzelthin S.I. Hille A. Pradier O. Conformal radiotherapy planning of cervix carcinoma: differences in the delineation of the clinical target volume. A comparison between gynaecologic and radiation oncologists Radiother Oncol 67 2003 87 95 10.1016/s0167-8140(02)00373-0 12758244
18 Tyng C.J. Chojniak R. Pinto P.N.V. Borba M.A. Bitencourt A.G.V. Fogaroli R.C. Conformal radiotherapy for lung cancer: interobservers’ variability in the definition of gross tumor volume between radiologists and radiotherapists Radiat Oncol 4 2009 28 10.1186/1748-717X-4-28 19653915
19 Weiss E. Hess C.F. The impact of gross tumor volume (GTV) and clinical target volume (CTV) definition on the total accuracy in radiotherapy theoretical aspects and practical experiences Strahlenther Onkol 179 2003 21 30 10.1007/s00066-003-0976-5 12540981
20 Foroudi F. Haworth A. Pangehel A. Wong J. Roxby P. Duchesne G. Inter-observer variability of clinical target volume delineation for bladder cancer using CT and cone beam CT J Med Imaging Radiat Oncol 53 2009 100 106 10.1111/j.1754-9485.2009.02044.x 19453535
21 Nishioka K. Shimizu S. Kinoshita R. Inoue T. Onodera S. Yasuda K. Evaluation of inter-observer variability of bladder boundary delineation on cone-beam CT Radiat Oncol 8 2013 185 10.1186/1748-717X-8-185 23879876
22 McNair H.A. Wiseman T. Joyce E. Peet B. Huddart R.A. International survey; current practice in On-line adaptive radiotherapy (ART) delivered using Magnetic Resonance Image (MRI) guidance Tech Innov Patient Supp Radiat Oncol 16 2020 1 9 10.1016/j.tipsro.2020.08.002
23 Adair Smith G. Dunlop A. Alexander S.E. Barnes H. Casey F. Chick J. Evaluation of therapeutic radiographer contouring for magnetic resonance image guided online adaptive prostate radiotherapy Radiother Oncol 180 2023 109457 10.1016/j.radonc.2022.109457
24 DaBlaCa Danish Bladder Cancer Group. Behandling og opfølgning af muskelinvasiv blærekræft 2020:1–33. https://www.dmcg.dk/siteassets/kliniske-retningslinjer–-skabeloner-og-vejledninger/kliniske-retningslinjer-opdelt-pa-dmcg/blarecancer/dablaca_muskelinvasiv_1_1_admgodk111120.pdf.
25 Grégoire V. Mackie T. De Neve W. Gospodarowicz M. Purdy J.A. van Herk M. ICRU Report 83 J ICRU 2010 10
26 Kim H.S. Park S.B. Lo S.S. Monroe J.I. Sohn J.W. Bidirectional local distance measure for comparing segmentations Med Phys 39 2012 6779 6790 10.1118/1.4754802 23127072
27 Grønborg C. Vestergaard A. Høyer M. Söhn M. Pedersen E.M. Petersen J.B. Intra-fractional bladder motion and margins in adaptive radiotherapy for urinary bladder cancer Acta Oncol 54 2015 1461 1466 10.3109/0284186X.2015.1062138 26313410
28 Nishioka K. Shimizu S. Shinohara N. Ito Y.M. Abe T. Maruyama S. Analysis of inter- and intra fractional partial bladder wall movement using implanted fiducial markers Radiat Oncol 12 2017 44 10.1186/s13014-017-0778-z 28249609
29 Kouwenhoven E. Giezen M. Struikmans H. Measuring the similarity of target volume delineations independent of the number of observers Phys Med Biol 54 2009 2863 2873 10.1088/0031-9155/54/9/018 19384002
30 Debois M. Oyen R. Maes F. Verswijvel G. Gatti G. Bosmans H. The contribution of magnetic resonance imaging to the three-dimensional treatment planning of localized prostate cancer Int J Radiat Oncol Biol Phys 45 1999 857 865 10.1016/S0360-3016(99)00288-6 10571190
31 Henke L.E. Fischer-Valuck B.W. Rudra S. Wan L. Samson P.S. Srivastava A. Prospective imaging comparison of anatomic delineation with rapid kV cone beam CT on a novel ring gantry radiotherapy device Radiother Oncol 178 2023 109428 10.1016/j.radonc.2022.11.017
32 Robar J.L. Cherpak A. MacDonald R.L. Yashayaeva A. McAloney D. McMaster N. Novel technology allowing cone beam computed tomography in 6 seconds: a patient study of comparative image quality Pract Radiat Oncol 14 2024 277 286 10.1016/j.prro.2023.10.014 37939844
