  Deformable object manipulation is a classical and challenging research area
in robotics. Compared with rigid object manipulation, this problem is more
complex due to the deformation properties including elastic, plastic, and
elastoplastic deformation. In this paper, we describe a new deformable object
manipulation method including soft contact simulation, manipulation learning,
and sim-to-real transfer. We propose a novel approach utilizing Vision-Based
Tactile Sensors (VBTSs) as the end-effector in simulation to produce
observations like relative position, squeezed area, and object contour, which
are transferable to real robots. For a more realistic contact simulation, a new
simulation environment including elastic, plastic, and elastoplastic
deformations is created. We utilize RL strategies to train agents in the
simulation, and expert demonstrations are applied for challenging tasks.
Finally, we build a real experimental platform to complete the sim-to-real
transfer and achieve a 90% success rate on difficult tasks such as cylinder and
sphere. To test the robustness of our method, we use plasticine of different
hardness and sizes to repeat the tasks including cylinder and sphere. The
experimental results show superior performances of deformable object
manipulation with the proposed method.
