
==== Front
Z Med Phys
Z Med Phys
Zeitschrift für Medizinische Physik
0939-3889
1876-4436
Elsevier

S0939-3889(22)00121-0
10.1016/j.zemedi.2022.11.004
Original Paper
Prospective risk analysis of the online-adaptive artificial intelligence-driven workflow using the Ethos treatment system
Wegener Sonja Wegener_S1@ukw.de
⁎
Exner Florian Exner_F@ukw.de

Weick Stefan Weick_S@ukw.de

Stark Silke Stark_S@ukw.de

Hutzel Heike Hutzel_H@ukw.de

Lutyj Paul Lutyj_P@ukw.de

Tamihardja Jörg Tamihardja_J@ukw.de
1
Razinskas Gary Razinskas_G@ukw.de
1
University of Wuerzburg, Department of Radiation Oncology, Wuerzburg, Germany
⁎ Corresponding author: Sonja Wegener, University of Wuerzburg, Department of Radiation Oncology, Josef-Schneider-Str. 11, 97080 Wuerzburg, Germany. Wegener_S1@ukw.de
1 Both authors contributed equally to this work.

09 12 2022
8 2024
09 12 2022
34 3 384396
8 7 2022
14 11 2022
© 2022 The Author(s)
2022
https://creativecommons.org/licenses/by-nc-nd/4.0/ This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Purpose

The recently introduced Varian Ethos system allows adjusting radiotherapy treatment plans to anatomical changes on a daily basis. The system uses artificial intelligence to speed up the process of creating adapted plans, comes with its own software solutions and requires a substantially different workflow. A detailed analysis of possible risks of the associated workflow is presented.

Methods

A prospective risk analysis of the adaptive workflow with the Ethos system was performed using Failure Modes and Effects Analysis (FMEA). An interprofessional team collected possible adverse events and evaluated their severity as well as their chance of occurrence and detectability. Measures to reduce the risks were discussed.

Results

A total of 122 events were identified, and scored. Within the 20 events with the highest-ranked risks, the following were identified: Challenges due to the stand-alone software solution with very limited connectivity to the existing record and verify software and digital patient file, unfamiliarity with the new software and its limitations and the adaption process relying on results obtained by artificial intelligence. The risk analysis led to the implementation of additional quality assurance measures in the workflow.

Conclusions

The thorough analysis of the risks associated with the new treatment technique was the basis for designing details of the workflow. The analysis also revealed challenges to be addressed by both, the vendor and customers. On the vendor side, this includes improving communication between their different software solutions. On the customer side, this especially includes establishing validation strategies to monitor the results of the black box adaption process making use of artificial intelligence.

Keywords

Risk analysis
Online adaptive radiotherapy
Ethos
Artificial intelligence
==== Body
pmc1 Introduction

Anatomical changes between or within fractions of a treatment series may lead to prepared treatment plans no longer creating the desired dose distribution with respect to the target volumes or organs at risk. Adaptive radiotherapy aims to modify plans to regard the current anatomy using a variety of tools [1], [2]. The Varian Ethos treatment system (Varian Medical Systems, Palo Alto, California, United States) enables online adaptive radiotherapy using artificial intelligence (AI) and machine learning based on kilovoltage (kV)2 images as described in detail in the literature [3]. Very briefly summarized, the image-guided adaptive radiotherapy (IGART) session consists of the following steps:• Acquisition of kV cone beam computed tomography (CBCT) images followed by detection of relevant organ structures called “influencers” by the software.

• Editing of the influencers by a trained physician or radiation therapist, called “Adaptor”.

• Propagation of the target structures based on an algorithm that considers the changes of the influencers.

• Validating or editing of the target structures by a specialist physician, called “Advanced Adaptor”.

• Creation of two treatment plans by the software, one being identical to the previously prepared plan of the patient recalculated on the anatomy of the day called “scheduled plan”, the other being re-optimized called the “adaptive plan”.

• Review of the treatment plan. When the adapted plan is chosen, a secondary dose calculation is performed as means of quality assurance (QA) before the treatment.

Despite the novelty of the Ethos system, treatments or emulator studies of different anatomical sites have been reported, including the pelvis region [4], bladder [5], lung [6] and head and neck [7].

It is recognized that the radiotherapy process consists of many complex steps with many potential sources of errors, which should be analyzed as a whole [8]. The literature provides several examples of risk analysis applied to the radiotherapy workflow as a whole [9], single aspects of the workflow such as automated contouring and treatment planning tools [10] or different treatment techniques, for example surface-guided deep inspiration breath-hold breast radiotherapy [11], stereotactic radiotherapy [12], total skin electron irradiation [13] and brachytherapy [14]. Risk analysis of online adaptive workflows has been carried out for radiotherapy combined with magnetic resonance imaging [15], [16] and quality assurance in the context of adaptive radiotherapy in general [17]. For the Halcyon linear accelerator, which shares the same hardware as the Ethos but otherwise follows the standard image-guided radiotherapy (IGRT) workflow, an analysis of the installation process [18] as well as a clinical safety assessment [19] is available. The newly introduced IGART workflow on the Ethos system contains components relying on AI. An assessment of the workflow of different treatment machines including Halcyon and Ethos indicated that additional causal scenarios that my lead to unintended or unexpected outcome are introduced with the adaptive workflow [19]. To our knowledge, this is the first detailed report of a risk analysis of the particular Ethos workflow based on Failure Modes and Effects Analysis (FMEA).

The introduction of a new treatment technique requires a risk analysis according to the German law (§126 StrSchV). In our case, the local authorities also demanded an elaborated report of the analysis results within a few weeks after the machine’s installation as a prerequisite for their approval. In the following, our way of organizing the process and the main results of the FME-analysis are presented. This could help other institutions considering the advantages or disadvantages of the same treatment technique or implementing the adaptive workflow with the same or a similar machine.

2 Material and methods

2.1 The process to be analyzed

The Ethos allows for both, standard IGRT and adaptive IGART treatments. While details of the IGART workflow will differ between institutions, the overall steps will be universal. The workflow analyzed in this report is as follows (Fig. 1): The first step includes all necessary preparations before the patient undergoes computed tomography (CT) for treatment planning purposes. Included herein are the booking of appointments, the initial examination of the patient, the decision for a treatment concept, as well as triggering the subsequent tasks using the electronic workflow management tool “ARIA’s Care Path” (Varian Medical Systems, Palo Alto, California, United States). Secondly, the planning CT for the initial treatment plan is acquired. Moreover, the adaption process at the treatment machine relies on the planning CT data for the generation of the synthetic CT. Thirdly, a preliminary prescription is generated, and the contours are delineated on the planning CT images. The choice of influencers and definition of the target structures affect the IGART sessions. A list of possible influencers per body region is provided by the vendor, but the user may omit some. Targets may either be derived from other structures or contoured directly. In the next step, a treatment plan is generated, reviewed and accepted. The planning requires a different mindset than IGRT planning: The quality of the achieved dose distribution during planning is just one aspect. In addition, the details of the stated goals for planning need to be robust, generating high-quality plans also on a similar but not identical anatomy during the adaption.Figure 1 Workflow for the adaptive treatment divided into 7 steps. The number of identified events is indicated for each activity in parentheses. Major changes between the adaptive and the standard IGRT workflow occurred in steps 2 to 6.

Step five then includes all preparations for the actual treatment, specifically the quality assurance (QA) of the treatment plan. In step six, the actual patient treatment is performed. Here, there are major differences to the IGRT workflow, as contouring of influencers and target volumes on the anatomy of the day is executed, and the adapted plan is checked by means of secondary dose calculation in every single fraction. At our institution, a trained radiation oncologist and a physicist need to be personally present at the machine for each IGART fraction due to regulations imposed by the local authorities. The last step then includes all activities after the final irradiation, including documentation, archiving and follow-up. While many of the activities in the IGART workflow shown above are the same as for IGRT patients, some new aspects are added or done differently using the stand-alone Ethos software.

2.2 Risk management team

The risk analysis was performed in an interdisciplinary team of four physicists, two radiation oncologists and two radiotherapists, eight members in total. Among the members were the physicist responsible for the machine, the physicist responsible for the software, the physicist responsible for planning, the senior physician in charge of the new treatment system and his first deputy, as well as two radiotherapists assigned to the machine for the first months. An additional physicist moderated the joint meetings. All, but the moderator, had attended the training week provided by the vendor. The team met only four times in total. Wherever possible, tasks were distributed to individuals to prepare until the next meeting, since the limited number of meetings required high efficiency.

2.3 Risk management approach

In general, the risk management process was organized to widely follow the recommendations of the Deutsche Gesellschaft für Medizinische Physik (DGMP) report number 25 [19]. A risk analysis covering the complete workflow of the department already existed for IGRT with the Halcyon. In order to prepare for the installation of the new machine, this analysis was updated in a team of similar composition a few months before the installation. That team specifically included the planning workflow in Eclipse, the treatment at the Halcyon accelerator, as well as work with the digital patient file in Aria into the analysis. On the Ethos machine, IGART and IGRT treatments are possible. Since the risk analysis of the IGRT workflow had large been covered already in the prior analysis of the Halcyon workflow, these aspects were excluded from the present analysis. The prior investigations allowed to reduce the range of topics to be covered by the adaptive team, concentrating on aspects unique to the IGART workflow as well as aspects requiring a reevaluation due to the use of new software or procedures. While the considered steps for the IGART workflow are visualized in Fig. 1, the whole IGRT workflow with the excluded steps is provided as a supplement. Not considered were the following activities: Patient registration and first appointment including medical history and examination, documentation, all details of patient positioning at the planning CT except those possibly affecting the image quality of the adaptive process, all details of patient positioning at the time of treatment except those possibly affecting the image quality of the adaptive process, the final appointment with the physician as well as follow-up.

FMEA was chosen as the risk analysis technique due to the team’s experience with this technique and its wide use and recognition. First, the overall workflow was divided into single steps, such as the initial Concept and Contouring, Planning and Adaptive Treatment. Each step was subdivided into different activities, assigning possible adverse events to these activities. The consequences of these events were discussed. For some of the events, it became necessary to further investigate details, which was carried out by individuals and reported at team meetings.

After completion of the list of events, individuals independently scored the severity S, occurrence O and detectability D of all the listed events that they felt comfortable to judge. Scoring was performed using any integer on a scale from one to five (see Table 1), which is adapted from the procedure described in [18], [19]. All personnel that had attended the vendor training participated in the evaluation. Nine scoring sheets were returned in total, five of these from radiation oncologists, three from physicist and one joint work from both participating therapists.Table 1 Scoring scale for severity, occurrence and detectability.

Score	Severity S	Occurrence O	Detectability D	
1	no or negligible consequences	> 5 years	>99.8%	
2	damage heals within a short time period	1-5 years	>99%	
3	damage requires prolongation of treatment with unclear results	1 month- 1 year	>95%	
4	permanent damage of a bodily function unavoidable	1 day – 1 month	>80%	
5	death immediately or within a short time period	< 1 day	<80%	

2.4 Evaluation

A risk priority number RPN was calculated from the averaged values over all submitted scores for each item for severity S, occurrence O and detectability D as:RPN=S∗O∗D.

Although the RPN has been criticized [20], [21], it is one of the methods appearing in the guidelines for radiotherapy risk analysis [8], [19]. The data were presented graphically in forms of histograms. Based on the mean RPN of 12.0 and the median RPN of 11.5 for the earlier analysis of the Halcyon IGRT workflow, it was decided to concentrate on events with RPN ≥ 11.5. The 20 highest-ranked events were discussed in the risk analysis team to provide possible solutions to the problems and include them in the standard operating procedures (SOPs) generated for the new workflow. To assess improvements, the same personnel participating in the initial scoring then scored the occurrence and detectability of these 20 events again individually. Seven scoring sheets were returned for calculation of updated RPN.

3 Results and discussion

3.1 Identified events and scoring results

A total of 122 events were identified which were not evenly distributed between the seven steps (Fig. 2). Most events were associated with Concept and Contouring (step 3) and Adaptive Treatment (step 6), while relatively few events were identified for Registration for Treatment (step 1) and Documentation and Follow-up (step 7). As only events due to the IGART workflow were considered, most events fall into the steps from preparation to the delivery of the actual treatment fraction, while few events are linked to the administrative workflow before the CT and after the final treatment session, which experienced very little changes.Figure 2 Number of possible events identified per step, 122 events in total.

S, O, D and the calculated RPN grouped by step are displayed in Fig. 3. Not only do steps 3 to 6 contain a larger number of events (Fig. 2), they also include the events with the highest RPNs. The highest recorded RPN was 20.8. The histogram of RPN over all events (Fig. 4) reveals that the majority of plans have RPN below 12. 16 events were assigned a RPN of 12 or above. However, it should be noted that the RPN just serves as a tool for ranking the events for further analysis and that the numbers are not proportional to the underlying risks.Figure 3 Boxplot diagram of (a) average severity, (b) occurrence and (c) detectability scored for each event separated into the different steps of the treatment process and (d) the calculated risk priority number. Explanation of the different steps see Fig. 1.

Figure 4 Histogram of the risk priority numbers RPN of the 122 evaluated events.

The consequences of the events can be pooled into six groups, resulting in wrong dose (number of events n = 27, average RPN = 10.5), quality loss in terms of a suboptimal plan or small dose deviation (n = 19, RPN = 10.3), wrong contours (n = 16, RPN = 9.4), expected issues with the artificial intelligence or image registration (n = 28, RPN = 8.8), time delay (n = 20, RPN = 6.9) or others not fitting into the previous groups (n = 12, RPN = 9.6).

Although there have been previous risk analyses of adaptive workflows [22], [15], [16], [17], no explicit analysis of the Ethos workflow has been presented. The analysis of a general adaptive radiotherapy workflow using FMEA also identified a high portion of failure modes concerning segmentation, the treatment planning process [17] or in segmentation, planning and the treatment beam delivery [16].

3.2 Top 20 events

The 20 events with the highest RPNs are listed in Table 2. Most of the highly ranked events are present in steps 6 (8 events) and 3 (6 events), being the Adaptive Treatment and Concept and Contouring.Table 2 20 highest-ranked events.

RPN rank	RPN	activity	Event	O	S	D	
1	20.8	5.5	Check of relevant plan parameters by physicists incomplete	2.8	2.8	2.8	
2	16.4	3.2	Overlap with prior radiation treatment not evaluated as dose sum is not displayed	2.8	2.2	2.7	
3	16.0	3.5	Important notes for the treatment machine not read as entered in wrong place	2.7	2.4	2.4	
4	15.4	6.2	Questionable dose distributions in regions with high density differences accompanied by large anatomic changes	2.6	3.0	2.0	
5	15.0	6.5	Images not checked in offline review as not notified about new images	2.1	2.6	2.7	
6	14.4	4.4	Isodose display confusing as not according to previous institution standards	2.3	3.1	2.0	
7	14.2	6.3	Patient details not known to Advanced Adaptor	2.7	2.3	2.3	
8	14.1	5.4	Point dose measurements using Mobius Verification Phantom do not provide enough information on the plan quality	2.5	2.5	2.3	
9	13.8	6.3	Mobius3D QA of wrong plan evaluated	2.5	2.0	2.8	
10	13.7	3.6	Old CT dataset used for treatment planning	2.7	1.9	2.7	
11	12.8	6.1	Patient is irradiated multiple times a day as appointment was not manually registered	3.0	1.8	2.3	
12	12.8	5.4	Mobius3D recalculation not sufficient to evaluate plan quality	2.3	3.3	1.8	
13	12.5	3.5	Prescriptions in Ethos software and Aria no longer consistent	2.5	2.5	2.0	
14	12.4	6.5	Images not independently checked	2.0	2.7	2.3	
15	12.3	3.3	Concept and Phases not correctly entered at initial step	2.2	2.8	2.0	
16	12.0	6.3	PTV not contoured correctly during the adaption process	3.0	2.0	2.0	
17	11.9	6.3	Influencer structures not contoured correctly during the adaption process	2.3	2.1	2.4	
18	11.8	7.3	Information on pre-irradiation not easily accessible	2.6	2.1	2.1	
19	11.5	3.5	Non-ideal parameters for planning in case there is no template available	2.3	2.7	1.9	
20	11.5	7.2	Loss (of accessibility) of treatment plans or dose distributions	3.0	1.6	2.4	

The events and our approach to mitigating the risk are individually described in the following.• Rank 1: Check of relevant plan parameters by physicists incomplete (RPN 20.8)

For the current IGRT patients, the check of the plan quality and technical parameters by the physicists before the first fraction is perceived as a crucial step in providing high-quality plans. While the relevant aspects to be checked are well known from long experience with the existing planning systems and included in checklists or in automatic scripts applied to finished plans, there is hardly any information on what may go wrong in the planning software for Ethos. Additionally, the IGART workflow may require attention to other aspects, such as robust or anticipatory optimization parameters also suitable for the new optimization during the adaptive session. An incomplete check of technical parameters can lead to over- or under-dosages for the patient. As a reaction, the generation of a list of items to be considered before giving the technical approval was initialized with the anticipated problems. This list needs to be updated regularly. (updated RPN: 18.5)• Rank 2: Overlap with prior radiation treatment not evaluated as dose sum is not displayed (RPN 16.4)

In the established planning systems, cumulative dose maps of previous and current irradiations can be created and visualized. These information are considered during the decision on a therapy concept. The Ethos software is not tailored to this purpose. Consequently, clinical decisions may be taken without the necessary information, leading to wrong concepts and doses. During the discussion, there was consensus that software limitations should not lead to less information being available. As a practicable procedure, all necessary dose summations will be created in another software after the planning process is finished but before the plan is presented for peer-review and prescribed. (updated RPN: 13.7)• Rank 3: Important notes for the treatment machine not read as entered in wrong place (RPN: 16.0)

Information needs to be transferred between different professions. The new software offers additional fields to enter information, especially a large field for the definition of the intention. These information cannot be easily edited later on and are not displayed at the treatment machine either. Crucial information entered here may not get noticed, resulting in a decreased outcome. The team realized that clear rules need to be established where to enter and retrieve information in the system, especially in combination with the digital patient file in another software. The results of the discussion were recorded in an unambiguous SOP. (updated RPN: 11.8)• Rank 4: Questionable dose distributions in regions with high density differences accompanied by large anatomic changes (RPN 15.4)

The CBCT acquired at the treatment machine in combination with the influencer structures is the basis for the synthetic CT. Density information is retrieved from the planning CT to generate the synthetic CT, on which the scheduled plan will then be recalculated, and the adapted plan will be optimized. First experiments with anthropomorphic phantoms including large inhomogeneities in combination with changes of the anatomy showed discrepancies between the densities observed in the CBCT and the synthetic CT, which led to a skepticism regarding these cases. As consequences, problems with the adaption process and with the dose distribution may arise in extreme cases, preventing treatments and leading to wrong dose distributions. As a reaction, we agreed to limit adaptive treatments to the rather homogeneous pelvic region until further investigations are completed. At the same time, we implemented additional quality control. After the first adaptive treatment and after any observed major changes, the dose distribution of the newly created adaptive plan is to be recalculated on the CBCT to reveal discrepancies between the synthetic CT and the CBCT. (updated RPN: 10.1)• Rank 5: Images not checked in offline review as not notified about new images (RPN 15.0)

In the IGRT workflow, the radiotherapists adjust the patient positioning according to the acquired images. Each matching is double-checked by a radiation oncologist shortly afterwards. Until now, new images to be checked appear in their task folder. The Ethos software does not allow filtering for unchecked images. Instead, patients in treatment need to be actively opened to look for new images. At the same time, it is not only the positioning that is adjusted during each fraction, but the complete targets and organs at risk are newly contoured. Having a second set of eyes double-check these contours seems even more important here, as a decrease of quality, including over- and underdosages, seem possible consequences. Two measures were taken: The therapists continuously accompanying the patients during the whole course of the treatment need to be involved in the contouring process assisting the physician. Their familiarity with the patient anatomy will help spot irregularities if a radiation oncologist not involved in the individual case needs to step in for organizational reasons. Additionally, images including contours are still to be verified by the treating radiation oncologists. In order to ensure that no images are forgotten, it is recommended to open the list of unchecked images at least on a weekly basis, here coinciding with the same interval as for the patient consultation. (updated RPN: 10.8)• Rank 6: Isodose display confusing as not according to previous institution standards (RPN 14.4)

For the so far established treatment planning systems, an identical color scheme for isodose lines had been implemented at our institution to facilitate the judgement of plans. The Ethos software allows a quick and flexible adjustment of displayed dose levels using a scroll bar, but displaying the isodose lines in the same fashion as used in our prior treatment planning systems is not possible. The team suspects that the unusual representation may lead to misinterpretations and wrong clinical decisions, resulting in mistakenly approved plans. As a first measure, a standard isodose display within the Ethos software was created. We suggested to Varian to enable a more flexible choice of displaying dose distributions. (updated RPN: 9.7)• Rank 7: Patient details not known to Advanced Adaptor (RPN 14.2)

The Advanced Adaptor, i.e. the radiation oncologist in our workflow, needs to verify and possibly modify the target proposed by the software during the adaption process. The radiation oncologist present at the treatment machine is usually not the radiation oncologist who was responsible for the initial contouring, who is aware of the details that led to the definition of the target volume. Consequently, the target may not be appropriately covered during the adaption process resulting in reduced target coverage. As a reaction, it was agreed that whenever possible the treating radiation oncologist should be present during the first fraction together with the radiation oncologist trained in the adaptive workflow. Important information should be fixated in a written form for universal access also in case of necessary personnel changes. An additional computer to display the initial plan in the Ethos software was positioned next to the treatment computer for easy reference. It is also evident that the aspect of individual contouring becomes less critical when a strict concept based on the definition of clinical target volumes is followed. (updated RPN: 12.7)• Rank 8: Point dose measurements using Mobius Verification Phantom do not provide enough information on the plan quality (RPN 14.1)

Although using the identical hardware as the Halcyon linear accelerator, the new Ethos software solution does not allow for the use of portal dosimetry. Instead, the software Mobius3D is part of the Ethos bundle and also comes with a pre-configured beam model ready for use. It allows secondary dose calculation, analysis of the machine log files and point dose measurements in a phantom. The evaluation of the correct application of the plan based on a single point dose was perceived as critical. It was decided to perform pre-treatment QA by measurements with a phantom providing multiple measurement points until the new Mobius3D software (Varian Medical Systems, Palo Alto, California, United States) including the inherent beam modelling was thoroughly evaluated. (updated RPN: 9.8)• Rank 9: Mobius3D QA of wrong plan evaluated (RPN 13.8)

During an adaptive treatment series, several plans are generated and exported to Mobius3D. There are typically at least four plans per fraction, two new ones generated each time the target volumes are edited. The plans are unambiguously labelled with number or letter identifiers. It is possible that the QA of the wrong plan version is evaluated during the adaptive treatment, such that a plan with insufficient quality is treated. As a measure, the SOP now includes that the radiation oncologist upon plan acceptance clearly indicates the ID of the plan and the number of monitor units verbally to double-check that the QA of the right plan variant is considered before the second signature is given and the treatment is continued. (updated RPN: 7.7)• Rank 10: Old CT dataset used for treatment planning (RPN 13.7)

During the import of CT images for contouring, data from different sources, including those from the picture archiving and communication system PACS, are displayed. In some cases, a large amount of data can be displayed, possibly choosing an old CT leading to a wrong dose distribution. Therefore, it was decided to import images through Eclipse only, to disable the display of the PACS data and leaving only a few CTs visible. The warning by the software, when an image older than eight weeks is selected, should not be dismissed easily. (updated RPN: 9.0)• Rank 11: Patient is irradiated multiple times a day as appointment was not manually registered (RPN 12.8)

In previous record-and-verify systems, the patient appointments were included in a treatment calendar. Open sessions were automatically closed when the treatment was successfully finished. As the treatment calendar for the Ethos machine is in Aria and as there is no direct communication between the treatment machine and Aria, all appointments need to be manually closed after treatment. Additionally, the Ethos software requires a definition of the time interval between treatments during the creation of the treatment intent, which cannot be altered later on without invalidation of the plan and re-optimization. If this interval is set to once a day, it is impossible to do two irradiations in one day, even in cases where patient circumstances or holidays require an exception. Therefore, we decided to make 6 hours the default interval. If the interval is set to a defined number of hours and the appointment is not manually finished after the treatment, the patient will be available again once the indicated time has passed. One possible event is that patients may be accidentally irradiated twice per day. To avoid this, the therapists close all appointments manually directly after the treatment, which is perceived as very prone to errors. As open appointments with a prior treatment within the stated period are displayed in grey, a good practice would be checking if such sessions are visible, e.g. in the middle of the day just before the lunchbreak. (updated RPN: 8.3)• Rank 12: Mobius3D recalculation not sufficient to evaluate plan quality (RPN 12.8)

The result of the secondary dose calculation in Mobius3D is the criterion for the technical acceptance of an adapted plan. Mobius3D is not completely independent, as it also uses the synthetic CT for the recalculation. This issue was already discussed in point 4. To facilitate work with Mobius3D, a table with the core parameters to consider when interpreting Mobius3D results for quick reference was created. This approach will be supported by a detailed analysis of the sensitivity and specificity of the Mobius3D software. (updated RPN: 11.8)• Rank 13: Prescriptions in Ethos software and Aria no longer consistent (RPN 12.5)

The prescription is done in Aria while the RT intent is entered in the Ethos software. As changes in the prescription do not invalidate the plans in Ethos, discrepancies are possible leading to a wrong number of treatments and documentation problems with consequences when the patient returns for a re-irradiation. Comparable to point 3, the only solution seemed the strict definition of where the valid prescription is entered and clear SOPs, which include the tasks necessary to map the changes of the prescription in Aria to the intent in the Ethos software. (updated RPN: 9.0)• Rank 14: Images not independently checked (RPN 12.3)

The point of missing double-checks of the treatment images and contouring was already discussed in combination with rank 5. (updated RPN: 7.8)• Rank 15: Concept and Phases not correctly entered at initial step (RPN 12.3)

In the Ethos software, a concept needs to be entered in the intent identifying different phases, for example two phases if a sequential boost is desired. We initially got the wrong information that all phases need to be entered initially and any amendments of phases would invalidate the current phase. A discussion of the point is now obsolete. The correct procedure to enter new phases was included in the SOP for the intent and contouring. (updated RPN: 4.2)• Rank 16: PTV not contoured correctly during the adaption process (RPN 12.0)

Target volumes may be wrongly contoured during the IGART sessions leading to inadequate plans. Consistent involvement of the therapist during the IGART sessions and checking the images (point 5) will help identify and prevent such events. (updated RPN: 11.9)• Rank 17: Influencer structures not contoured correctly during the adaption process (RPN 11.9)

Contouring of the influencer is a critical step, as the synthetic CT and the suggested target volumes are based on the changes of these contours. Wrong influencers may lead to reduced quality of the derived target structures, increased time for the adaption process, or even over- and underdosages. Similar to the concept of mutual control employed in point 5, the therapist and radiation oncologist should both be attentive during the contouring. (updated RPN: 9.9)• Rank 18: Information on pre-irradiation not easily accessible (RPN 11.8)

Sessions treated with Ethos are recorded in the Ethos Treatment Management Software and are not visible in Aria. It is therefore not possible to get information on the number of treated fractions from Aria. This gets even more complicated if some fractions are treated on a machine that records in Aria, such as the Halcyon. Consequences may be a wrong number of fractions treated in total and a chaotic documentation. This point was ranked quite low, as the previous experience with our Halcyon promised almost no downtime, but in fact already occurred during the first week of treatment due to major connectivity and server problems. After thorough consideration, it was decided to manually simulate each fraction treated on Ethos in Aria to have a complete record at all times. The simulation is done by the therapists directly after each fraction, and the completeness is checked by the physicist on a daily basis. Additionally, a detailed SOP and a CarePath were created to describe how to transfer information between Ethos and Halcyon. The procedure is very time-consuming and risky, so we limited changes of the treatment machine to unavoidable exceptions only. In the interest of safety, we think that this issue should be fixed in a future software version. (updated RPN: 7.9)• Rank 19: Non-ideal parameters for planning in case there is no template available (RPN 11.5)

Planning templates with goals were prepared for standard cases. When no such template is available for the plan type, the goals are to be entered in a way that a plan of high quality and adequate for adaption is created. Small details, such as the priority assigned to a goal, decide whether the dose-volume histogram information will be visible at the treatment machine. Creating a good plan without a template requires much experience. Other attempts at planning may result in lower quality dose distributions or problems during the adaption. It was agreed that all planning needs to be carried out in a team of a physicist and a radiation oncologist. Wherever possible, templates should be used. In order to create the necessary templates, the physics group needs to be informed about the interest in the irradiation of new body regions well in advance. (updated RPN: 7.0)• Rank 20: Loss (of accessibility) of treatment plans or dose distributions (RPN 11.5)

The storage of treatment plans was discussed under the aspect of data loss. The initial plan report is documented as a PDF. The dose distributions of the individual sessions are accumulated within the software, but no report of the final patient dose is generated. Due to software problems or issues with archiving, the dose distribution may not be accessible at a later time, when the patient returns for further radiation treatments. The consequences may be a delay or wrong concept decisions at future treatments. It was agreed that the dose accumulation displayed in the software should be reviewed weekly to be able to react to systematic problems. Further, the final accumulated dose distribution after the last fraction is to be documented as a PDF and loaded into the patient file. (updated RPN: 13.2)

The list of the top 20 risks illustrates that there is the necessity for strict protocols to be followed. One of the main tasks evolving from the risk analysis was developing SOPs including all the discussed issues. This is consistent with the mitigation measures identified for adaptive treatments using magnetic-resonance-guided adaptive radiotherapy. Klüter et al. [15] also reported that a standardized workflow with clear-defined protocols was of high importance. Many issues arise only because the Ethos software is encapsulated from the digital patient file in Aria. An integrated solution or improved communication between the two software solutions, but also an easy transferability between Halcyon and Ethos is ultimately desirable. Some minor points, such as further display options for isodose lines, would be helpful. What remains is the critical investigation of the capability and limitation of the adaption process in combination with the use of AI.

The proposed measures aim to reduce either occurrence or detectability or both. This reduces the RPN in all but one of the top 20 events, with a median 3.5 reduction, indicating an assumed efficacy of the implemented measures. Although measures were identified to reduce the occurrence of loss of accessibility of the treatment plans or dose distributions (rank 20), its RPN rose in the re-evaluation. This is likely caused by several of the team members increasing their occurrence scores due to increased awareness of problems with data storage following the common discussion. We will further investigate additional measures.

The Ethos Treatment Management software, both a treatment planning system and a record-and-verify system, was a main focus of the analysis. This is reflected in approximately a third of all events being connected to either step 3 Concept and Contouring or step 4 Planning (Fig. 2). In terms of the calculation algorithm, Ethos as well as Eclipse use Acuros XB, which have been shown to yield comparable dosimetric accuracy [22]. Upon acceptance of the treatment machine, the data was validated and a comparison of the two Linacs and the treatment planning systems (Ethos Treatment Management, calculation during the adaption process, Eclipse) was performed. All components are regularly checked for consistency. In addition to the QA performed on the Halcyon system, an adaptive check is performed that ensures the correct dose to the correct position in a phantom irradiation following the IGART workflow.

For plan-specific QA, during adaptive sessions only a secondary dose calculation is performed (compare events ranked 8, 9 and 12). Nevertheless, our workflow still relies on measurement-based pre-treatment QA. Since the Ethos software does not support the evaluation of portal dosimetry neither pre-treatment nor in vivo, the implemented procedure differs from the standard on our Halcyon. For pre-treatment QA, we decided to stick to ArcCHECK measurements until we thoroughly evaluated the Mobius3D software including the provided measurement phantom (compare event ranked 8). For IGART QA of the adapted plans, additional verification calculations after the treatment sessions were implemented. Based on the results of our analysis and the discussed mitigation measures, we agree with the opinion in previous work [15], [17] that the implementation or development of additional QA measures for adaptive radiotherapy is one of the necessities for safe adaptive radiotherapy, also for IGART.

The sum of the RPN of the Top 20 events for the IGART workflow is 275.2 compared to 525.9 for the IGRT workflow from the previous Halcyon risk analysis. The highest IGART RPN of 20.8 (Fig. 4) is smaller than IGRT rank 20 with an RPN of 22.8. At first sight, it may seem surprising that the new IGART workflow is associated with lower risks. This is the consequence of a lower mean value for all three scores of S, O and D for the IGART compared to the IGRT workflow in the Top 20. Of course, this is partly due to different staff scoring at the two occasions. One hypothesis for why especially O and D dropped is the teaming up of a radiation oncologist and a physicist during the planning as well as during each treatment session, such that they may detect errors and correct each other immediately. This is in contrast to the general analysis presented by Noel et al. [17] of an adaptive workflow: There, detection probability was assumed to decrease due to time constraints, user inattention or inadequate training. In the IGRT workflow Top 20, several of the events are due to communication issues not perceived to play a role while having dedicated staff and plenty of time for only a few patients treated with the special IGART technique during its introductory phase. A main challenge will be the roll-out to large patient numbers should IGART become the standard procedure.

3.3 Limitations and Outlook

Any prospective risk analysis using FMEA relies on the assumptions taken for severity S, occurrence O and detectability D. Upon the introduction of a new procedure, the occurrence can only be estimated from prior experience in comparable situations, limited experience during the vendor training and measurements during the installation phase. Therefore, FMEA relies on the subjective scores provided by individuals. The example of the event ranked as number 3 illustrates the variability of the scoring: 7 scores were submitted with S ranging from 1 to 4, O ranging from 2 to 4 and D ranging from 1 to 4. This led to individual RPNs between 8 and 48 for different scorers. The threshold value to accept an RPN is also arbitrary to some extent. Here, the mean value of the prior risk analysis of the Halcyon workflow was used as the threshold for further discussion of the possible events. The extensive recording of critical incidents or irregularities in the workflow during the first year with the new treatment technique was highly encouraged within the department, to get more quantitative data for the re-evaluation of the workflow. It is also evident that upon introduction of the new treatment technique, the lists of possible events that the team compiled is in no way complete. Likewise, the severity was estimated, although the exact consequences are often difficult to predict for this system using components relying on artificial intelligence. The thorough evaluation of the parameters influencing its work was identified as one of the main challenges to be addressed in future research. For these reasons, the team agreed on performing a second risk analysis to reevaluate the workflow after one year.

The time frame for the analysis was very tight and needed to take place in the few weeks between the initial training by the vendor and mostly before the treatment of the first patients. Additionally, the local authorities expected the final report of the analysis results about one month after completion of the training. Therefore, the team only met four times for sessions of around 60 to 90 minutes each time. The first three sessions were dedicated to compiling the list of steps in the workflow and events. In order to work more efficiently, each step was prepared for discussion by a member of the most involved profession. Preparation included highlighting the steps where the IGART workflow was different from the institution’s IGRT workflow or where new software was used, such that the events needed to be amended, scored and evaluated. During the sessions, the team added further events and discussed the consequences. Between the third and the fourth meeting, all scoring and the analysis of the results were carried out to provide the list of 20 highest-ranked events for the final meeting. This procedure helped stick to a reasonable time frame and limited resources as far as possible. Having just updated the overall workflow in the institution further limited the risk analysis to the aspects relevant for the new treatment technique.

FMEA using the RPN is a common tool [8], [19], but has its limitations [20], [21]. For example, a seldom-occurring event with high severity may have a similar RPN to a common event with a small severity. Being aware of this limitation, we also regarded the events with a severity of 3 or more. They are either included in the top 20 (rank 11, rank 16, rank 20) or similar in type that they have already been covered with the measures implemented for the highest-ranked events.

Only 20 highest-ranked risks were discussed in the interprofessional team. However, other points were also considered when the standard operating procedures for the various tasks were established within smaller groups or even within single professions. Being at the start of the introduction of the new process, only some members of staff were acquainted with the workflow and trained by the vendor, such that the number of possible scorers was limited and not everyone was able to send in their scores within the timeframe. For a re-evaluation, more feedback from all professions is desirable.

One of the main achievements of the risk analysis was the common discussion of possible failures, which improved the awareness of the most crucial steps in the process among all participants. Changes to the initially proposed workflow were agreed upon in the team and subsequently implemented. Staffing the risk analysis team with members having large influence on the procedure paid off. Members from the team were also responsible for writing up the SOPs for the individual steps and could independently incorporate the discussed points immediately.

4 Conclusions

Moving from IGRT to adaptive IGART treatments revealed over 100 additional or changed possible adverse events. The analysis contributed to a better understanding of the associated risks, allowed refining of the workflow, and revealed further research and evaluation questions.

The consequences of the analysis were that precise SOPs were formulated to define everyone’s roles and responsibilities in a complex process. Some software and connectivity functions of Ethos and Aria not being available led to some of the highest-ranked risks. Although the hardware for Halcyon and Ethos are comparable, the Ethos software stands for itself and the limited communication between Ethos Treatment Management and Aria necessitates a lot of manual documentation, which is both tedious and prone to errors. Ethos utilizes AI for the planning process and the generation of contours during the adaption process. The system often appears to be a black box. A priori, it is not clear which aspects influence the quality of the adaption process, so several aspects that require further investigation were identified.

For teams carrying out a similar analysis, we recommend the creation or an update of the whole radiotherapy chain shortly before the introduction of the new system, such that the analysis of the new treatment technique can focus on the new aspects and is manageable in a short time period. Additionally, we profited from having all those involved in creating the workflow and writing the SOPs present, such that decisions could be taken during the sessions and proposed changes could quickly be implemented.

CRediT authorship contribution statement

Sonja Wegener: Conceptualization, Methodology, Investigation, Writing – original draft. Florian Exner: Conceptualization, Methodology, Project administration, Writing – review & editing. Stefan Weick: Investigation, Writing – review & editing. Silke Stark: Investigation. Heike Hutzel: Investigation. Paul Lutyj: Investigation. Jörg Tamihardja: Investigation, Writing – review & editing. Gary Razinskas: Investigation, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A Supplementary data

The following are the Supplementary data to this article:Supplementary figure 1

Acknowledgements

The authors thank Klaus Bratengeier for contributing to the discussions during the team sessions, Frederick Mantel, Ingulf Lawrenz and Marcus Zimmermann for participating in the scoring and Otto A. Sauer for critical comments on the manuscript.

Appendix A Supplementary data to this article can be found online at https://doi.org/10.1016/j.zemedi.2022.11.004.

2 Abbreviations: AI – artificial intelligence, CT – computed tomography - D – detectability, DGMP – Deutsche Gesellschaft für Medizinische Physik, FMEA – failure-mode-and-effects analysis, IGART – image-guided adaptive radiotherapy, IGRT – image-guided radiotherapy, kV – kilovoltage, O – occurrence, QA – quality assurance, RPN – risk priority number, S – severity, QA – quality assurance.
==== Refs
References

1 Green O.L. Henke L.E. Hugo G.D. Practical Clinical Workflows for Online and Offline Adaptive Radiation Therapy Semin Radiat Oncol 29 3 2019 219 227 31027639
2 Thörnqvist S. Adaptive radiotherapy strategies for pelvic tumors – a systematic review of clinical implementations Acta Oncol 55 8 2016 943 958 27055486
3 Archambault Y. Making on-line adaptive radiotherapy possible using artificial intelligence and machine learning for efficient daily re-planning. Medical Physics Int J 8 2 2020
4 Sibolt P. Clinical implementation of artificial intelligence-driven cone-beam computed tomography-guided online adaptive radiotherapy in the pelvic region Phys Imag Radiat Oncol 17 2020 1 7
5 Åström L.M. Online adaptive radiotherapy of urinary bladder cancer with full re-optimization to the anatomy of the day: Initial experience and dosimetric benefits Radiother Oncol 171 2022 37 42 35358605
6 Mao W. Evaluation of Auto-Contouring and Dose Distributions for Online Adaptive Radiation Therapy of Patients With Locally Advanced Lung Cancers Pract Radiat Oncol 2022
7 Yoon S.W. Initial Evaluation of a Novel Cone-Beam CT-Based Semi-Automated Online Adaptive Radiotherapy System for Head and Neck Cancer Treatment - A Timing and Automation Quality Study Cureus 12 8 2020 e9660 32923257
8 Huq M.S. The report of Task Group 100 of the AAPM: Application of risk analysis methods to radiation therapy quality management Med Phys 43 7 2016 4209 4262 27370140
9 Lohmann D. Risk analysis for radiotherapy at the Universitätsklinikum Erlangen Z Med Phys 2022
10 Nealon K.A. Using Failure Mode and Effects Analysis to Evaluate Risk in the Clinical Adoption of Automated Contouring and Treatment Planning Tools Pract Radiat Oncol 2022
11 Bright M. Failure modes and effects analysis for surface-guided DIBH breast radiotherapy J Appl Clin Med Phys 23 4 2022 e13541 35112445
12 Veronese I. Multi-institutional application of Failure Mode and Effects Analysis (FMEA) to CyberKnife Stereotactic Body Radiation Therapy (SBRT) Radiat Oncol 10 1 2015 132 26071401
13 Shariff M. Dosimetry, Optimization and FMEA of Total Skin Electron Irradiation (TSEI) Z Med Phys 2021
14 Mayadev J. A failure modes and effects analysis study for gynecologic high-dose-rate brachytherapy Brachytherapy 14 6 2015 866 875 26204807
15 Klüter S. A practical implementation of risk management for the clinical introduction of online adaptive Magnetic Resonance-guided radiotherapy Phys Imag Radiat Oncol 17 2021 53 57
16 Nishioka S. Identifying risk characteristics using failure mode and effect analysis for risk management in online magnetic resonance-guided adaptive radiation therapy Phys Imag Radiat Oncol 23 2022 1 7
17 Noel C.E. Process-based quality management for clinical implementation of adaptive radiotherapy Med Phys 41 8Part1 2014 081717
18 Teo P.T. Application of TG-100 risk analysis methods to the acceptance testing and commissioning process of a Halcyon linear accelerator Med Phys 46 3 2019 1341 1354 30620406
19 Gließmann S, Hoinkis C, Knak A, Kornek D, Lange A, Milickovic N. DGMP-Bericht Nr. 25, Eine Prozessbeschreibung zur Umsetzung des Risikomanagements für die Strahlenbehandlung gemäß §126 StrlSchV; 2022.
20 Buchgeister M. Hummel D. Risikoanalyse in der Strahlentherapie: Muss es die FMEA-Methode mit RPZ sein? Z Med Phys 31 4 2021 343 345 34657782
21 Braband J. MIT NEUEM ANSATZ ZU VERLÄSSLICHEN RISIKOPRIORITÄTSZAHLEN Beschränktes Risiko QZ Qualität und Zuverlässigkeit 2 2008 28 33
22 Hu Y. Validation of the preconfigured Varian Ethos Acuros XB Beam Model for treatment planning dose calculations: A dosimetric study J Appl Clin Med Phys 21 12 2020 27 42
