  Reinforcement Learning (RL) has been witnessed its potential for training a
dialogue policy agent towards maximizing the accumulated rewards given from
users. However, the reward can be very sparse for it is usually only provided
at the end of a dialog session, which causes unaffordable interaction
requirements for an acceptable dialog agent. Distinguished from many efforts
dedicated to optimizing the policy and recovering the reward alternatively
which suffers from easily getting stuck in local optima and model collapse, we
decompose the adversarial training into two steps: 1) we integrate a
pre-trained language model as a discriminator to judge whether the current
system action is good enough for the last user action (i.e., \textit{next
action prediction}); 2) the discriminator gives and extra local dense reward to
guide the agent's exploration. The experimental result demonstrates that our
method significantly improves the complete rate (~4.4\%) and success rate
(~8.0\%) of the dialogue system.
