  Although many pretrained models exist for text or images, there have been
relatively fewer attempts to train representations specifically for dialog
understanding. Prior works usually relied on finetuned representations based on
generic text representation models like BERT or GPT-2. But such language
modeling pretraining objectives do not take the structural information of
conversational text into consideration. Although generative dialog models can
learn structural features too, we argue that the structure-unaware word-by-word
generation is not suitable for effective conversation modeling. We empirically
demonstrate that such representations do not perform consistently across
various dialog understanding tasks. Hence, we propose a structure-aware Mutual
Information based loss-function DMI (Discourse Mutual Information) for training
dialog-representation models, that additionally captures the inherent
uncertainty in response prediction. Extensive evaluation on nine diverse dialog
modeling tasks shows that our proposed DMI-based models outperform strong
baselines by significant margins.
