  Recent work on language modelling has shifted focus from count-based models
to neural models. In these works, the words in each sentence are always
considered in a left-to-right order. In this paper we show how we can improve
the performance of the recurrent neural network (RNN) language model by
incorporating the syntactic dependencies of a sentence, which have the effect
of bringing relevant contexts closer to the word being predicted. We evaluate
our approach on the Microsoft Research Sentence Completion Challenge and show
that the dependency RNN proposed improves over the RNN by about 10 points in
accuracy. Furthermore, we achieve results comparable with the state-of-the-art
models on this task.
