  Large Language Models (LLMs), although powerful in general domains, often
perform poorly on domain-specific tasks like medical question answering (QA).
Moreover, they tend to function as "black-boxes," making it challenging to
modify their behavior. To address the problem, our study delves into retrieval
augmented generation (RAG), aiming to improve LLM responses without the need
for fine-tuning or retraining. Specifically, we propose a comprehensive
retrieval strategy to extract medical facts from an external knowledge base,
and then inject them into the query prompt for LLMs. Focusing on medical QA
using the MedQA-SMILE dataset, we evaluate the impact of different retrieval
models and the number of facts provided to the LLM. Notably, our
retrieval-augmented Vicuna-7B model exhibited an accuracy improvement from
44.46% to 48.54%. This work underscores the potential of RAG to enhance LLM
performance, offering a practical approach to mitigate the challenges of
black-box LLMs.
