  Open-domain dialogue generation suffers from the data insufficiency problem
due to the vast size of potential responses. In this paper, we propose to
explore potential responses by counterfactual reasoning. Given an observed
response, the counterfactual reasoning model automatically infers the outcome
of an alternative policy that could have been taken. The resulting
counterfactual response synthesized in hindsight is of higher quality than the
response synthesized from scratch. Training on the counterfactual responses
under the adversarial learning framework helps to explore the high-reward area
of the potential response space. An empirical study on the DailyDialog dataset
shows that our approach significantly outperforms the HRED model as well as the
conventional adversarial learning approaches.
