  Traditional Korean medicine (TKM) emphasizes individualized diagnosis and
treatment. This uniqueness makes AI modeling difficult due to limited data and
implicit processes. Large language models (LLMs) have demonstrated impressive
medical inference, even without advanced training in medical texts. This study
assessed the capabilities of GPT-4 in TKM, using the Korean National Licensing
Examination for Korean Medicine Doctors (K-NLEKMD) as a benchmark. The
K-NLEKMD, administered by a national organization, encompasses 12 major
subjects in TKM. We optimized prompts with Chinese-term annotation, English
translation for questions and instruction, exam-optimized instruction, and
self-consistency. GPT-4 with optimized prompts achieved 66.18% accuracy,
surpassing both the examination's average pass mark of 60% and the 40% minimum
for each subject. The gradual introduction of language-related prompts and
prompting techniques enhanced the accuracy from 51.82% to its maximum accuracy.
GPT-4 showed low accuracy in subjects including public health &
medicine-related law, internal medicine (2) which are localized in Korea and
TKM. The model's accuracy was lower for questions requiring TKM-specialized
knowledge. It exhibited higher accuracy in diagnosis-based and recall-based
questions than in intervention-based questions. A positive correlation was
observed between the consistency and accuracy of GPT-4's responses. This study
unveils both the potential and challenges of applying LLMs to TKM. These
findings underline the potential of LLMs like GPT-4 in culturally adapted
medicine, especially TKM, for tasks such as clinical assistance, medical
education, and research. But they also point towards the necessity for the
development of methods to mitigate cultural bias inherent in large language
models and validate their efficacy in real-world clinical settings.
