  Even though reinforcement-learning-based algorithms achieved superhuman
performance in many domains, the field of robotics poses significant challenges
as the state and action spaces are continuous, and the reward function is
predominantly sparse. Furthermore, on many occasions, the agent is devoid of
access to any form of demonstration. Inspired by human learning, in this work,
we propose a method named highlight experience replay (HiER) that creates a
secondary highlight replay buffer for the most relevant experiences. For the
weights update, the transitions are sampled from both the standard and the
highlight experience replay buffer. It can be applied with or without the
techniques of hindsight experience replay (HER) and prioritized experience
replay (PER). Our method significantly improves the performance of the
state-of-the-art, validated on 8 tasks of three robotic benchmarks.
Furthermore, to exploit the full potential of HiER, we propose HiER+ in which
HiER is enhanced with an arbitrary data collection curriculum learning method.
Our implementation, the qualitative results, and a video presentation are
available on the project site: http://www.danielhorvath.eu/hier/.
