  Reinforcement Learning (RL) algorithms can suffer from poor sample efficiency
when rewards are delayed and sparse. We introduce a solution that enables
agents to learn temporally extended actions at multiple levels of abstraction
in a sample efficient and automated fashion. Our approach combines universal
value functions and hindsight learning, allowing agents to learn policies
belonging to different time scales in parallel. We show that our method
significantly accelerates learning in a variety of discrete and continuous
tasks.
