  Contact-rich manipulation tasks are commonly found in modern manufacturing
settings. However, manually designing a robot controller is considered hard for
traditional control methods as the controller requires an effective combination
of modalities and vastly different characteristics. In this paper, we firstly
consider incorporating operational space visual and haptic information into
reinforcement learning(RL) methods to solve the target uncertainty problem in
unstructured environments. Moreover, we propose a novel idea of introducing a
proactive action to solve the partially observable Markov decision process
problem. Together with these two ideas, our method can either adapt to
reasonable variations in unstructured environments and improve the sample
efficiency of policy learning. We evaluated our method on a task that involved
inserting a random-access memory using a torque-controlled robot, and we tested
the success rates of the different baselines used in the traditional methods.
We proved that our method is robust and can tolerate environmental variations
very well.
