  Exploration in environments with sparse feedback remains a challenging
research problem in reinforcement learning (RL). When the RL agent explores the
environment randomly, it results in low exploration efficiency, especially in
robotic manipulation tasks with high dimensional continuous state and action
space. In this paper, we propose a novel method, called Augmented
Curiosity-Driven Experience Replay (ACDER), which leverages (i) a new
goal-oriented curiosity-driven exploration to encourage the agent to pursue
novel and task-relevant states more purposefully and (ii) the dynamic initial
states selection as an automatic exploratory curriculum to further improve the
sample-efficiency. Our approach complements Hindsight Experience Replay (HER)
by introducing a new way to pursue valuable states. Experiments conducted on
four challenging robotic manipulation tasks with binary rewards, including
Reach, Push, Pick&Place and Multi-step Push. The empirical results show that
our proposed method significantly outperforms existing methods in the first
three basic tasks and also achieves satisfactory performance in multi-step
robotic task learning.
