  Large language models can now directly generate answers to many factual
questions without referencing external sources. Unfortunately, relatively
little attention has been paid to methods for evaluating the quality and
correctness of these answers, for comparing the performance of one model to
another, or for comparing one prompt to another. In addition, the quality of
generated answers are rarely directly compared to the quality of retrieved
answers. As models evolve and prompts are modified, we have no systematic way
to measure improvements without resorting to expensive human judgments. To
address this problem we adapt standard retrieval benchmarks to evaluate answers
generated by large language models. Inspired by the BERTScore metric for
summarization, we explore two approaches. In the first, we base our evaluation
on the benchmark relevance judgments. We empirically run experiments on how
information retrieval relevance judgments can be utilized as an anchor to
evaluating the generated answers. In the second, we compare generated answers
to the top results retrieved by a diverse set of retrieval models, ranging from
traditional approaches to advanced methods, allowing us to measure improvements
without human judgments. In both cases, we measure the similarity between an
embedded representation of the generated answer and an embedded representation
of a known, or assumed, relevant passage from the retrieval benchmark.
