  The quickly-expanding nature of published medical literature makes it
challenging for clinicians and researchers to keep up with and summarize
recent, relevant findings in a timely manner. While several closed-source
summarization tools based on large language models (LLMs) now exist, rigorous
and systematic evaluations of their outputs are lacking. Furthermore, there is
a paucity of high-quality datasets and appropriate benchmark tasks with which
to evaluate these tools. We address these issues with four contributions: we
release Clinfo.ai, an open-source WebApp that answers clinical questions based
on dynamically retrieved scientific literature; we specify an information
retrieval and abstractive summarization task to evaluate the performance of
such retrieval-augmented LLM systems; we release a dataset of 200 questions and
corresponding answers derived from published systematic reviews, which we name
PubMed Retrieval and Synthesis (PubMedRS-200); and report benchmark results for
Clinfo.ai and other publicly available OpenQA systems on PubMedRS-200.
