  Goal-oriented reinforcement learning has recently been a practical framework
for robotic manipulation tasks, in which an agent is required to reach a
certain goal defined by a function on the state space. However, the sparsity of
such reward definition makes traditional reinforcement learning algorithms very
inefficient. Hindsight Experience Replay (HER), a recent advance, has greatly
improved sample efficiency and practical applicability for such problems. It
exploits previous replays by constructing imaginary goals in a simple heuristic
way, acting like an implicit curriculum to alleviate the challenge of sparse
reward signal. In this paper, we introduce Hindsight Goal Generation (HGG), a
novel algorithmic framework that generates valuable hindsight goals which are
easy for an agent to achieve in the short term and are also potential for
guiding the agent to reach the actual goal in the long term. We have
extensively evaluated our goal generation algorithm on a number of robotic
manipulation tasks and demonstrated substantially improvement over the original
HER in terms of sample efficiency.
