  Deep Reinforcement Learning (DRL) enables robots to perform some intelligent
tasks end-to-end. However, there are still many challenges for long-horizon
sparse-reward robotic manipulator tasks. On the one hand, a sparse-reward
setting causes exploration inefficient. On the other hand, exploration using
physical robots is of high cost and unsafe. In this paper, we propose a method
of learning long-horizon sparse-reward tasks utilizing one or more existing
traditional controllers named base controllers in this paper. Built upon Deep
Deterministic Policy Gradients (DDPG), our algorithm incorporates the existing
base controllers into stages of exploration, value learning, and policy update.
Furthermore, we present a straightforward way of synthesizing different base
controllers to integrate their strengths. Through experiments ranging from
stacking blocks to cups, it is demonstrated that the learned state-based or
image-based policies steadily outperform base controllers. Compared to previous
works of learning from demonstrations, our method improves sample efficiency by
orders of magnitude and improves the performance. Overall, our method bears the
potential of leveraging existing industrial robot manipulation systems to build
more flexible and intelligent controllers.
