  Actor-critic (AC) algorithms are a class of model-free deep reinforcement
learning algorithms, which have proven their efficacy in diverse domains,
especially in solving continuous control problems. Improvement of exploration
(action entropy) and exploitation (expected return) using more efficient
samples is a critical issue in AC algorithms. A basic strategy of a learning
algorithm is to facilitate indiscriminately exploring all of the environment
state space, as well as to encourage exploring rarely visited states rather
than frequently visited one. Under this strategy, we propose a new method to
boost exploration through an intrinsic reward, based on measurement of a
state's novelty and the associated benefit of exploring the state (with regards
to policy optimization), altogether called plausible novelty. With incentivized
exploration of plausible novel states, an AC algorithm is able to improve its
sample efficiency and hence training performance. The new method is verified by
extensive simulations of continuous control tasks of MuJoCo environments on a
variety of prominent off-policy AC algorithms.
