  We study multi-turn response generation for open-domain dialogues. The
existing state-of-the-art addresses the problem with deep neural architectures.
While these models improved response quality, their complexity also hinders the
application of the models in real systems. In this work, we pursue a model that
has a simple structure yet can effectively leverage conversation contexts for
response generation. To this end, we propose four auxiliary tasks including
word order recovery, utterance order recovery, masked word recovery, and masked
utterance recovery, and optimize the objectives of these tasks together with
maximizing the likelihood of generation. By this means, the auxiliary tasks
that relate to context understanding can guide the learning of the generation
model to achieve a better local optimum. Empirical studies with three
benchmarks indicate that our model can significantly outperform
state-of-the-art generation models in terms of response quality on both
automatic evaluation and human judgment, and at the same time enjoys a much
faster decoding process.
