  This paper aims to establish an entropy-regularized value-based reinforcement
learning method that can ensure the monotonic improvement of policies at each
policy update. Unlike previously proposed lower-bounds on policy improvement in
general infinite-horizon MDPs, we derive an entropy-regularization aware lower
bound. Since our bound only requires the expected policy advantage function to
be estimated, it is scalable to large-scale (continuous) state-space problems.
We propose a novel reinforcement learning algorithm that exploits this
lower-bound as a criterion for adjusting the degree of a policy update for
alleviating policy oscillation. We demonstrate the effectiveness of our
approach in both discrete-state maze and continuous-state inverted pendulum
tasks using a linear function approximator for value estimation.
