  As large language models (LLMs) develop increasingly sophisticated
capabilities and find applications in medical settings, it becomes important to
assess their medical safety due to their far-reaching implications for personal
and public health, patient safety, and human rights. However, there is little
to no understanding of the notion of medical safety in the context of LLMs, let
alone how to evaluate and improve it. To address this gap, we first define the
notion of medical safety in LLMs based on the Principles of Medical Ethics set
forth by the American Medical Association. We then leverage this understanding
to introduce MedSafetyBench, the first benchmark dataset specifically designed
to measure the medical safety of LLMs. We demonstrate the utility of
MedSafetyBench by using it to evaluate and improve the medical safety of LLMs.
Our results show that publicly-available medical LLMs do not meet standards of
medical safety and that fine-tuning them using MedSafetyBench improves their
medical safety. By introducing this new benchmark dataset, our work enables a
systematic study of the state of medical safety in LLMs and motivates future
work in this area, thereby mitigating the safety risks of LLMs in medicine.
