  Before deploying a language model (LM) within a given domain, it is important
to measure its tendency to generate factually incorrect information in that
domain. Existing methods for factuality evaluation of LLM generation focus on
facts sampled from the LM itself, and thus do not control the set of evaluated
facts and might under-represent domain specific or rare facts. We propose
FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for
evaluating LM factuality. FACTOR automatically transforms a factual corpus of
interest into a benchmark evaluating an LM's propensity to generate true facts
from the corpus vs. similar but incorrect statements. We use our framework to
create three benchmarks: Wiki-FACTOR, News-FACTOR and Expert-FACTOR. We show
that: (i) our benchmark scores increase with model size and improve when the LM
is augmented with retrieval; (ii) benchmark score and perplexity do not always
agree on model ranking; (iii) when perplexity and benchmark score disagree, the
latter better reflects factuality in open-ended generation, as measured by
human annotators. We make our data and code publicly available in
https://github.com/AI21Labs/factor.
