  Multi-goal robot manipulation tasks with sparse rewards are difficult for
reinforcement learning (RL) algorithms due to the inefficiency in collecting
successful experiences. Recent algorithms such as Hindsight Experience Replay
(HER) expedite learning by taking advantage of failed trajectories and
replacing the desired goal with one of the achieved states so that any failed
trajectory can be utilized as a contribution to learning. However, HER
uniformly chooses failed trajectories, without taking into account which ones
might be the most valuable for learning. In this paper, we address this problem
and propose a novel approach Contact Energy Based Prioritization~(CEBP) to
select the samples from the replay buffer based on rich information due to
contact, leveraging the touch sensors in the gripper of the robot and object
displacement. Our prioritization scheme favors sampling of contact-rich
experiences, which are arguably the ones providing the largest amount of
information. We evaluate our proposed approach on various sparse reward robotic
tasks and compare them with the state-of-the-art methods. We show that our
method surpasses or performs on par with those methods on robot manipulation
tasks. Finally, we deploy the trained policy from our method to a real Franka
robot for a pick-and-place task. We observe that the robot can solve the task
successfully. The videos and code are publicly available at:
https://erdiphd.github.io/HER_force
