  We achieved contact-rich flexible object manipulation, which was difficult to
control with vision alone. In the unzipping task we chose as a validation task,
the gripper grasps the puller, which hides the bag state such as the direction
and amount of deformation behind it, making it difficult to obtain information
to perform the task by vision alone. Additionally, the flexible fabric bag
state constantly changes during operation, so the robot needs to dynamically
respond to the change. However, the appropriate robot behavior for all bag
states is difficult to prepare in advance. To solve this problem, we developed
a model that can perform contact-rich flexible object manipulation by real-time
prediction of vision with tactility. We introduced a point-based attention
mechanism for extracting image features, softmax transformation for predicting
motions, and convolutional neural network for extracting tactile features. The
results of experiments using a real robot arm revealed that our method can
realize motions responding to the deformation of the bag while reducing the
load on the zipper. Furthermore, using tactility improved the success rate from
56.7% to 93.3% compared with vision alone, demonstrating the effectiveness and
high performance of our method.
