  This paper presents a Discriminative Deep Dyna-Q (D3Q) approach to improving
the effectiveness and robustness of Deep Dyna-Q (DDQ), a recently proposed
framework that extends the Dyna-Q algorithm to integrate planning for
task-completion dialogue policy learning. To obviate DDQ's high dependency on
the quality of simulated experiences, we incorporate an RNN-based discriminator
in D3Q to differentiate simulated experience from real user experience in order
to control the quality of training data. Experiments show that D3Q
significantly outperforms DDQ by controlling the quality of simulated
experience used for planning. The effectiveness and robustness of D3Q is
further demonstrated in a domain extension setting, where the agent's
capability of adapting to a changing environment is tested.
