ESTRO 2024 - Abstract Book

S13 ESTRO 2024 In the talk, we will discuss ways to implement AI auto-segmentation algorithms in large-scale clinical practice. Since the availability of AI algorithms is rapidly increasing, we will focus on flexible and adaptable systems to be able to implement new and updated algorithms quickly. We will discuss procedures for validation before the introduction of an algorithm and how to incorporate the results of the validation into the daily use of the specific algorithm, quality assurance and verification of the complete treatment chain (End-to-end test), and continuous monitoring of AI-segmented structures after clinical implementation. Continuous monitoring has multiple purposes. Detecting the lack of editing of the automated contours is just as important as detecting systematic errors because it elucidates changes in the use of the algorithms. During the presentation, we will share our experiences with the introduction of AI algorithms and how it has changed work procedures with a focus on how to catch errors in an AI-derived structure. Finally, we will discuss educational needs and methods to ensure the continued ability of staff to identify correct as well as incorrect performance of AI algorithms. Invited Speaker

3296

Auto-segmentation QA

Liesbeth Vandewinckele

KU Leuven, Laboratory of Experimental Radiotherapy, Leuven, Belgium

Abstract:

Auto-segmentation is the most important application of Artificial Intelligence (AI) in radiotherapy. It is currently offered by a lot of companies and being introduced in a lot of radiotherapy departments. However, the quality assurance (QA) of auto-segmentation models remains challenging since the applied techniques appear as black boxes to the user. In the following, different types of QA are explained and recommendations are given of how to deal with them in clinical practice. Routine model QA is QA of the auto-segmentation model itself by evaluating the model output after a change in the clinical workflow. These changes can for example be software updates or changes in imaging device (CT, MRI, ...), imaging protocol (patient positioning, field of view, fixation aids), ... The evaluation of the model should in these cases be performed by an end-to-end test on a reference dataset that reflects the current clinical practice and has not been used to create the model. The obtained automatic segmentations should be compared in both a quantitative and qualitative manner to manually obtained segmentations. Quantitative metrics used for autosegmentation purposes are for example the DICE similarity coefficient, the Hausdorff distance or volume metrics. A qualitative evaluation should be performed by a physician by comparing both segmentations side by side. When the routine QA tests fail, re-commissioning of the autosegmentation model is necessary. Case-specific QA refers to the QA of the auto-segmented contours of an individual patient. Manual supervision of the obtained automatic segmentations is at this moment the most important tool for case-specific QA. The segmentations can in this way be adapted to the user's preference. Next to manual verifications, methods exist to flag outlier patients/segmentations upfront to the manual check. A first method is to perform a similarity check to compare the new patient's data to the ones of the training set since the behaviour of the model in situations it has not seen before is unknown. A second method consists of a statistical model that can detect outliers by evaluating volume, centroid and structure shape of the auto-segmentations. A third method is to simultaneously use an independent, secondary automatic segmentation methods that can reveal outliers if differences are present in both auto-segmentations.