  In this work, we propose contextual language models that incorporate dialog
level discourse information into language modeling. Previous works on
contextual language model treat preceding utterances as a sequence of inputs,
without considering dialog interactions. We design recurrent neural network
(RNN) based contextual language models that specially track the interactions
between speakers in a dialog. Experiment results on Switchboard Dialog Act
Corpus show that the proposed model outperforms conventional single turn based
RNN language model by 3.3% on perplexity. The proposed models also demonstrate
advantageous performance over other competitive contextual language models.
