  Large language models, such as GPT-4 and Med-PaLM, have shown impressive
performance on clinical tasks; however, they require access to compute, are
closed-source, and cannot be deployed on device. Mid-size models such as
BioGPT-large, BioMedLM, LLaMA 2, and Mistral 7B avoid these drawbacks, but
their capacity for clinical tasks has been understudied. To help assess their
potential for clinical use and help researchers decide which model they should
use, we compare their performance on two clinical question-answering (QA)
tasks: MedQA and consumer query answering. We find that Mistral 7B is the best
performing model, winning on all benchmarks and outperforming models trained
specifically for the biomedical domain. While Mistral 7B's MedQA score of 63.0%
approaches the original Med-PaLM, and it often can produce plausible responses
to consumer health queries, room for improvement still exists. This study
provides the first head-to-head assessment of open source mid-sized models on
clinical tasks.
