  In English and other languages, multiple adjectives in a complex noun phrase
show intricate ordering patterns that have been a target of much linguistic
theory. These patterns offer an opportunity to assess the ability of language
models (LMs) to learn subtle rules of language involving factors that cross the
traditional divisions of syntax, semantics, and pragmatics. We review existing
hypotheses designed to explain Adjective Order Preferences (AOPs) in humans and
develop a setup to study AOPs in LMs: we present a reusable corpus of adjective
pairs and define AOP measures for LMs. With these tools, we study a series of
LMs across intermediate checkpoints during training. We find that all models'
predictions are much closer to human AOPs than predictions generated by factors
identified in theoretical linguistics. At the same time, we demonstrate that
the observed AOPs in LMs are strongly correlated with the frequency of the
adjective pairs in the training data and report limited generalization to
unseen combinations. This highlights the difficulty in establishing the link
between LM performance and linguistic theory. We therefore conclude with a road
map for future studies our results set the stage for, and a discussion of key
questions about the nature of knowledge in LMs and their ability to generalize
beyond the training sets.
