  In recent years, there have been significant breakthroughs in the field of
natural language processing, particularly with the development of large
language models (LLMs). These LLMs have showcased remarkable capabilities on
various benchmarks. In the healthcare field, the exact role LLMs and other
future AI models will play remains unclear. There is a potential for these
models in the future to be used as part of adaptive physician training, medical
co-pilot applications, and digital patient interaction scenarios. The ability
of AI models to participate in medical training and patient care will depend in
part on their mastery of the knowledge content of specific medical fields. This
study investigated the medical knowledge capability of LLMs, specifically in
the context of internal medicine subspecialty multiple-choice test-taking
ability. We compared the performance of several open-source LLMs (Koala 7B,
Falcon 7B, Stable-Vicuna 13B, and Orca Mini 13B), to GPT-4 and Claude 2 on
multiple-choice questions in the field of Nephrology. Nephrology was chosen as
an example of a particularly conceptually complex subspecialty field within
internal medicine. The study was conducted to evaluate the ability of LLM
models to provide correct answers to nephSAP (Nephrology Self-Assessment
Program) multiple-choice questions. The overall success of open-sourced LLMs in
answering the 858 nephSAP multiple-choice questions correctly was 17.1% -
25.5%. In contrast, Claude 2 answered 54.4% of the questions correctly, whereas
GPT-4 achieved a score of 73.3%. We show that current widely used open-sourced
LLMs do poorly in their ability for zero-shot reasoning when compared to GPT-4
and Claude 2. The findings of this study potentially have significant
implications for the future of subspecialty medical training and patient care.
