  Deep reinforcement learning has recently gained a focus on problems where
policy or value functions are independent of goals. Evidence exists that the
sampling of goals has a strong effect on the learning performance, but there is
a lack of general mechanisms that focus on optimizing the goal sampling
process. In this work, we present a simple and general goal masking method that
also allows us to estimate a goal's difficulty level and thus realize a
curriculum learning approach for deep RL. Our results indicate that focusing on
goals with a medium difficulty level is appropriate for deep deterministic
policy gradient (DDPG) methods, while an "aim for the stars and reach the
moon-strategy", where hard goals are sampled much more often than simple goals,
leads to the best learning performance in cases where DDPG is combined with for
hindsight experience replay (HER). We demonstrate that the approach
significantly outperforms standard goal sampling for different robotic object
manipulation problems.
