  Accurate estimation of the relative pose between an object and a robot hand
is critical for many manipulation tasks. However, most of the existing
object-in-hand pose datasets use two-finger grippers and also assume that the
object remains fixed in the hand without any relative movements, which is not
representative of real-world scenarios. To address this issue, a 6D
object-in-hand pose dataset is proposed using a teleoperation method with an
anthropomorphic Shadow Dexterous hand. Our dataset comprises RGB-D images,
proprioception and tactile data, covering diverse grasping poses, finger
contact states, and object occlusions. To overcome the significant hand
occlusion and limited tactile sensor contact in real-world scenarios, we
propose PoseFusion, a hybrid multi-modal fusion approach that integrates the
information from visual and tactile perception channels. PoseFusion generates
three candidate object poses from three estimators (tactile only, visual only,
and visuo-tactile fusion), which are then filtered by a SelectLSTM network to
select the optimal pose, avoiding inferior fusion poses resulting from modality
collapse. Extensive experiments demonstrate the robustness and advantages of
our framework. All data and codes are available on the project website:
https://elevenjiang1.github.io/ObjectInHand-Dataset/
