  Large language models (LLMs) have made significant progress in various
domains, including healthcare. However, the specialized nature of clinical
language understanding tasks presents unique challenges and limitations that
warrant further investigation. In this study, we conduct a comprehensive
evaluation of state-of-the-art LLMs, namely GPT-3.5, GPT-4, and Bard, within
the realm of clinical language understanding tasks. These tasks span a diverse
range, including named entity recognition, relation extraction, natural
language inference, semantic textual similarity, document classification, and
question-answering. We also introduce a novel prompting strategy,
self-questioning prompting (SQP), tailored to enhance LLMs' performance by
eliciting informative questions and answers pertinent to the clinical scenarios
at hand. Our evaluation underscores the significance of task-specific learning
strategies and prompting techniques for improving LLMs' effectiveness in
healthcare-related tasks. Additionally, our in-depth error analysis on the
challenging relation extraction task offers valuable insights into error
distribution and potential avenues for improvement using SQP. Our study sheds
light on the practical implications of employing LLMs in the specialized domain
of healthcare, serving as a foundation for future research and the development
of potential applications in healthcare settings.
