  Large language models, particularly GPT-3, are able to produce high quality
summaries of general domain news articles in few- and zero-shot settings.
However, it is unclear if such models are similarly capable in more
specialized, high-stakes domains such as biomedicine. In this paper, we enlist
domain experts (individuals with medical training) to evaluate summaries of
biomedical articles generated by GPT-3, given zero supervision. We consider
both single- and multi-document settings. In the former, GPT-3 is tasked with
generating regular and plain-language summaries of articles describing
randomized controlled trials; in the latter, we assess the degree to which
GPT-3 is able to \emph{synthesize} evidence reported across a collection of
articles. We design an annotation scheme for evaluating model outputs, with an
emphasis on assessing the factual accuracy of generated summaries. We find that
while GPT-3 is able to summarize and simplify single biomedical articles
faithfully, it struggles to provide accurate aggregations of findings over
multiple documents. We release all data and annotations used in this work.
