  Transformer language models have received widespread public attention, yet
their generated text is often surprising even to NLP researchers. In this
survey, we discuss over 250 recent studies of English language model behavior
before task-specific fine-tuning. Language models possess basic capabilities in
syntax, semantics, pragmatics, world knowledge, and reasoning, but these
capabilities are sensitive to specific inputs and surface features. Despite
dramatic increases in generated text quality as models scale to hundreds of
billions of parameters, the models are still prone to unfactual responses,
commonsense errors, memorized text, and social biases. Many of these weaknesses
can be framed as over-generalizations or under-generalizations of learned
patterns in text. We synthesize recent results to highlight what is currently
known about large language model capabilities, thus providing a resource for
applied work and for research in adjacent fields that use language models.
