  Deep reinforcement learning (RL) has been endowed with high expectations in
tackling challenging manipulation tasks in an autonomous and self-directed
fashion. Despite the significant strides made in the development of
reinforcement learning, the practical deployment of this paradigm is hindered
by at least two barriers, namely, the engineering of a reward function and
ensuring the safety guaranty of learning-based controllers. In this paper, we
address these challenging limitations by proposing a framework that merges a
reinforcement learning \lstinline[columns=fixed]{planner} that is trained using
sparse rewards with a model predictive controller (MPC)
\lstinline[columns=fixed]{actor}, thereby offering a safe policy. On the one
hand, the RL \lstinline[columns=fixed]{planner} learns from sparse rewards by
selecting intermediate goals that are easy to achieve in the short term and
promising to lead to target goals in the long term. On the other hand, the MPC
\lstinline[columns=fixed]{actor} takes the suggested intermediate goals from
the RL \lstinline[columns=fixed]{planner} as the input and predicts how the
robot's action will enable it to reach that goal while avoiding any obstacles
over a short period of time. We evaluated our method on four challenging
manipulation tasks with dynamic obstacles and the results demonstrate that, by
leveraging the complementary strengths of these two components, the agent can
solve manipulation tasks in complex, dynamic environments safely with a $100\%$
success rate. Videos are available at
\url{https://videoviewsite.wixsite.com/mpc-hgg}.
