  Prior work has shown that, on small amounts of training data, syntactic
neural language models learn structurally sensitive generalisations more
successfully than sequential language models. However, their computational
complexity renders scaling difficult, and it remains an open question whether
structural biases are still necessary when sequential models have access to
ever larger amounts of training data. To answer this question, we introduce an
efficient knowledge distillation (KD) technique that transfers knowledge from a
syntactic language model trained on a small corpus to an LSTM language model,
hence enabling the LSTM to develop a more structurally sensitive representation
of the larger training data it learns from. On targeted syntactic evaluations,
we find that, while sequential LSTMs perform much better than previously
reported, our proposed technique substantially improves on this baseline,
yielding a new state of the art. Our findings and analysis affirm the
importance of structural biases, even in models that learn from large amounts
of data.
