  Solving goal-oriented tasks is an important but challenging problem in
reinforcement learning (RL). For such tasks, the rewards are often sparse,
making it difficult to learn a policy effectively. To tackle this difficulty,
we propose a new approach called Policy Continuation with Hindsight Inverse
Dynamics (PCHID). This approach learns from Hindsight Inverse Dynamics based on
Hindsight Experience Replay, enabling the learning process in a self-imitated
manner and thus can be trained with supervised learning. This work also extends
it to multi-step settings with Policy Continuation. The proposed method is
general, which can work in isolation or be combined with other on-policy and
off-policy algorithms. On two multi-goal tasks GridWorld and FetchReach, PCHID
significantly improves the sample efficiency as well as the final performance.
