  This position paper describes and critiques the Pretraining-Agnostic
Identically Distributed (PAID) evaluation paradigm, which has become a central
tool for measuring progress in natural language understanding. This paradigm
consists of three stages: (1) pre-training of a word prediction model on a
corpus of arbitrary size; (2) fine-tuning (transfer learning) on a training set
representing a classification task; (3) evaluation on a test set drawn from the
same distribution as that training set. This paradigm favors simple, low-bias
architectures, which, first, can be scaled to process vast amounts of data, and
second, can capture the fine-grained statistical properties of a particular
data set, regardless of whether those properties are likely to generalize to
examples of the task outside the data set. This contrasts with humans, who
learn language from several orders of magnitude less data than the systems
favored by this evaluation paradigm, and generalize to new tasks in a
consistent way. We advocate for supplementing or replacing PAID with paradigms
that reward architectures that generalize as quickly and robustly as humans.
