  Pragmatics and non-literal language understanding are essential to human
communication, and present a long-standing challenge for artificial language
models. We perform a fine-grained comparison of language models and humans on
seven pragmatic phenomena, using zero-shot prompting on an expert-curated set
of English materials. We ask whether models (1) select pragmatic
interpretations of speaker utterances, (2) make similar error patterns as
humans, and (3) use similar linguistic cues as humans to solve the tasks. We
find that the largest models achieve high accuracy and match human error
patterns: within incorrect responses, models favor literal interpretations over
heuristic-based distractors. We also find preliminary evidence that models and
humans are sensitive to similar linguistic cues. Our results suggest that
pragmatic behaviors can emerge in models without explicitly constructed
representations of mental states. However, models tend to struggle with
phenomena relying on social expectation violations.
