  Sparse rewards pose a significant challenge to achieving high sample
efficiency in goal-conditioned reinforcement learning (RL). Specifically, in
sequential manipulation tasks, the agent receives failure rewards until it
successfully completes the entire manipulation task, which leads to low sample
efficiency. To tackle this issue and improve sample efficiency, we propose a
novel model-based RL framework called Model-based Relay Hindsight Experience
Replay (MRHER). MRHER breaks down a continuous task into subtasks with
increasing complexity and utilizes the previous subtask to guide the learning
of the subsequent one. Instead of using Hindsight Experience Replay (HER) in
every subtask, we design a new robust model-based relabeling method called
Foresight relabeling (FR). FR predicts the future trajectory of the hindsight
state and relabels the expected goal as a goal achieved on the virtual future
trajectory. By incorporating FR, MRHER effectively captures more information
from historical experiences, leading to improved sample efficiency,
particularly in object-manipulation environments. Experimental results
demonstrate that MRHER exhibits state-of-the-art sample efficiency in benchmark
tasks, outperforming RHER by 13.79% and 14.29% in the FetchPush-v1 environment
and FetchPickandPlace-v1 environment, respectively.
