  Deep reinforcement learning (DRL) algorithms have successfully been
demonstrated on a range of challenging decision making and control tasks. One
dominant component of recent deep reinforcement learning algorithms is the
target network which mitigates the divergence when learning the Q function.
However, target networks can slow down the learning process due to delayed
function updates. Our main contribution in this work is a self-regularized
TD-learning method to address divergence without requiring a target network.
Additionally, we propose a self-guided policy improvement method by combining
policy-gradient with zero-order optimization to search for actions associated
with higher Q-values in a broad neighborhood. This makes learning more robust
to local noise in the Q function approximation and guides the updates of our
actor network. Taken together, these components define GRAC, a novel
self-guided and self-regularized actor critic algorithm. We evaluate GRAC on
the suite of OpenAI gym tasks, achieving or outperforming state of the art in
every environment tested.
