  We propose a exploration mechanism of policy in Deep Reinforcement Learning,
which is exploring more when agent needs, called Add Noise to Noise (AN2N). The
core idea is: when the Deep Reinforcement Learning agent is in a state of poor
performance in history, it needs to explore more. So we use cumulative rewards
to evaluate which past states the agents have not performed well, and use
cosine distance to measure whether the current state needs to be explored more.
This method shows that the exploration mechanism of the agent's policy is
conducive to efficient exploration. We combining the proposed exploration
mechanism AN2N with Deep Deterministic Policy Gradient (DDPG), Soft
Actor-Critic (SAC) algorithms, and apply it to the field of continuous control
tasks, such as halfCheetah, Hopper, and Swimmer, achieving considerable
improvement in performance and convergence speed.
