  Introduction: Recently, the effectiveness of Large Language Models (LLMs) has
increased rapidly, allowing them to be used in a great number of applications.
However, the risks posed by the generation of false information through LLMs
significantly limit their applications in sensitive areas such as healthcare,
highlighting the necessity for rigorous validations to determine their utility
and reliability. To date, no study has extensively compared the performance of
LLMs on Polish medical examinations across a broad spectrum of specialties on a
very large dataset. Objectives: This study evaluated the performance of three
Generative Pretrained Transformer (GPT) models on the Polish Board
Certification Exam (Pa\'nstwowy Egzamin Specjalizacyjny, PES) dataset, which
consists of 297 tests. Methods: We developed a software program to download and
process PES exams and tested the performance of GPT models using OpenAI
Application Programming Interface. Results: Our findings reveal that GPT-3.5
did not pass any of the analyzed exams. In contrast, the GPT-4 models
demonstrated the capability to pass the majority of the exams evaluated, with
the most recent model, gpt-4-0125, successfully passing 222 (75%) of them. The
performance of the GPT models varied significantly, displaying excellence in
exams related to certain specialties while completely failing others.
Conclusions: The significant progress and impressive performance of LLM models
hold great promise for the increased application of AI in the field of medicine
in Poland. For instance, this advancement could lead to the development of
AI-based medical assistants for healthcare professionals, enhancing the
efficiency and accuracy of medical services.
