  Languages typically provide more than one grammatical construction to express
certain types of messages. A speaker's choice of construction is known to
depend on multiple factors, including the choice of main verb -- a phenomenon
known as \emph{verb bias}. Here we introduce DAIS, a large benchmark dataset
containing 50K human judgments for 5K distinct sentence pairs in the English
dative alternation. This dataset includes 200 unique verbs and systematically
varies the definiteness and length of arguments. We use this dataset, as well
as an existing corpus of naturally occurring data, to evaluate how well recent
neural language models capture human preferences. Results show that larger
models perform better than smaller models, and transformer architectures (e.g.
GPT-2) tend to out-perform recurrent architectures (e.g. LSTMs) even under
comparable parameter and training settings. Additional analyses of internal
feature representations suggest that transformers may better integrate specific
lexical information with grammatical constructions.
