  Reinforcement Learning(RL) with sparse rewards is a major challenge. We
propose \emph{Hindsight Trust Region Policy Optimization}(HTRPO), a new RL
algorithm that extends the highly successful TRPO algorithm with
\emph{hindsight} to tackle the challenge of sparse rewards. Hindsight refers to
the algorithm's ability to learn from information across goals, including ones
not intended for the current task. HTRPO leverages two main ideas. It
introduces QKL, a quadratic approximation to the KL divergence constraint on
the trust region, leading to reduced variance in KL divergence estimation and
improved stability in policy update. It also presents Hindsight Goal
Filtering(HGF) to select conductive hindsight goals. In experiments, we
evaluate HTRPO in various sparse reward tasks, including simple benchmarks,
image-based Atari games, and simulated robot control. Ablation studies indicate
that QKL and HGF contribute greatly to learning stability and high performance.
Comparison results show that in all tasks, HTRPO consistently outperforms both
TRPO and HPG, a state-of-the-art algorithm for RL with sparse rewards.
