  We deploy the methods of controlled psycholinguistic experimentation to shed
light on the extent to which the behavior of neural network language models
reflects incremental representations of syntactic state. To do so, we examine
model behavior on artificial sentences containing a variety of syntactically
complex structures. We test four models: two publicly available LSTM sequence
models of English (Jozefowicz et al., 2016; Gulordava et al., 2018) trained on
large datasets; an RNNG (Dyer et al., 2016) trained on a small, parsed dataset;
and an LSTM trained on the same small corpus as the RNNG. We find evidence that
the LSTMs trained on large datasets represent syntactic state over large spans
of text in a way that is comparable to the RNNG, while the LSTM trained on the
small dataset does not or does so only weakly.
