  For in-hand manipulation, estimation of the object pose inside the hand is
one of the important functions to manipulate objects to the target pose. Since
in-hand manipulation tends to cause occlusions by the hand or the object
itself, image information only is not sufficient for in-hand object pose
estimation. Multiple modalities can be used in this case, the advantage is that
other modalities can compensate for occlusion, noise, and sensor malfunctions.
Even though deciding the utilization rate of a modality (referred to as
reliability value) corresponding to the situations is important, the manual
design of such models is difficult, especially for various situations. In this
paper, we propose deep gated multi-modal learning, which self-determines the
reliability value of each modality through end-to-end deep learning. For the
experiments, an RGB camera and a GelSight tactile sensor were attached to the
parallel gripper of the Sawyer robot, and the object pose changes were
estimated during grasping. A total of 15 objects were used in the experiments.
In the proposed model, the reliability values of the modalities were determined
according to the noise level and failure of each modality, and it was confirmed
that the pose change was estimated even for unknown objects.
