  Multi-goal reinforcement learning (RL) aims to qualify the agent to
accomplish multi-goal tasks, which is of great importance in learning scalable
robotic manipulation skills. However, reward engineering always requires
strenuous efforts in multi-goal RL. Moreover, it will introduce inevitable bias
causing the suboptimality of the final policy. The sparse reward provides a
simple yet efficient way to overcome such limits. Nevertheless, it harms the
exploration efficiency and even hinders the policy from convergence. In this
paper, we propose a density-based curriculum learning method for efficient
exploration with sparse rewards and better generalization to desired goal
distribution. Intuitively, our method encourages the robot to gradually broaden
the frontier of its ability along the directions to cover the entire desired
goal space as much and quickly as possible. To further improve data efficiency
and generality, we augment the goals and transitions within the allowed region
during training. Finally, We evaluate our method on diversified variants of
benchmark manipulation tasks that are challenging for existing methods.
Empirical results show that our method outperforms the state-of-the-art
baselines in terms of both data efficiency and success rate.
