ESTRO 2024 - Abstract Book

S2924

Interdiscplinary - Other

ESTRO 2024

The Aim of this study authors were to examined the capability of the Chat GPT model (ver 3.5) to identify the correct answer and the passing rate of the Chat GPT version 3.5 model in solving the vocational exam for Electroradiology Technician Technologist (qualification Z.21 and MS-19) in both English and Polish leanguage.

Material/Methods:

Obtaining questions from the Z.21 and MS.19 competency tests conducted from January 2015 to 2023. Each of these questions was presented to an A.I (chat GPT 3.5) model to generate answers. The questions were also translated into English. To assess the quality and performance of the AI model, the generated responses were analyzed using descriptive statistics. Obtained from the AI model results were compared with data from reports published by the Central Examination Commission, Qualifying Examination in a given year.

Results:

The highest and lowest percentages of correct answers were observed for different observers and in varying language contexts:

Highest Percentages of Proper Answers:

• In the Polish language, the model achieved percentages ranging from 77.7% to 80%, with the best performance observed by different observers on different occasions. • In English, the model displayed notably higher success rates, with percentages reaching as high as 92.59%.

Lowest Percentages of Proper Answers:

• The lowest success rates, ranging from 53% to 54%, were observed for Polish language questions. • The lowest percentages of correct answers in English ranged from 55% to 58%.

These findings indicate that the AI model's proficiency in answering questions is influenced by the language in which the questions are presented and the specific observer involved.

Conclusion:

In summary, the AI model demonstrated variable performance in answering questions from electroradiology technician professional exams, with generally higher accuracy when questions were presented in English. This suggests that the model's proficiency in answering questions is influenced by the language in which the questions are posed.

Keywords: GPT , RTT, Electroradiologists