  In this paper, we propose Inverse Adversarial Training (IAT) algorithm for
training neural dialogue systems to avoid generic responses and model dialogue
history better. In contrast to standard adversarial training algorithms, IAT
encourages the model to be sensitive to the perturbation in the dialogue
history and therefore learning from perturbations. By giving higher rewards for
responses whose output probability reduces more significantly when dialogue
history is perturbed, the model is encouraged to generate more diverse and
consistent responses. By penalizing the model when generating the same response
given perturbed dialogue history, the model is forced to better capture
dialogue history and generate more informative responses. Experimental results
on two benchmark datasets show that our approach can better model dialogue
history and generate more diverse and consistent responses. In addition, we
point out a problem of the widely used maximum mutual information (MMI) based
methods for improving the diversity of dialogue response generation models and
demonstrate it empirically.
