  A dialogue policy module is an essential part of task-completion dialogue
systems. Recently, increasing interest has focused on reinforcement learning
(RL)-based dialogue policy. Its favorable performance and wise action decisions
rely on an accurate estimation of action values. The overestimation problem is
a widely known issue of RL since its estimate of the maximum action value is
larger than the ground truth, which results in an unstable learning process and
suboptimal policy. This problem is detrimental to RL-based dialogue policy
learning. To mitigate this problem, this paper proposes a dynamic partial
average estimator (DPAV) of the ground truth maximum action value. DPAV
calculates the partial average between the predicted maximum action value and
minimum action value, where the weights are dynamically adaptive and
problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue
policy and show that our method can achieve better or comparable results
compared to top baselines on three dialogue datasets of different domains with
a lower computational load. In addition, we also theoretically prove the
convergence and derive the upper and lower bounds of the bias compared with
those of other methods.
