  When a toddler is presented a new toy, their instinctual behaviour is to pick
it upand inspect it with their hand and eyes in tandem, clearly searching over
its surface to properly understand what they are playing with. At any instance
here, touch provides high fidelity localized information while vision provides
complementary global context. However, in 3D shape reconstruction, the
complementary fusion of visual and haptic modalities remains largely
unexplored. In this paper, we study this problem and present an effective
chart-based approach to multi-modal shape understanding which encourages a
similar fusion vision and touch information.To do so, we introduce a dataset of
simulated touch and vision signals from the interaction between a robotic hand
and a large array of 3D objects. Our results show that (1) leveraging both
vision and touch signals consistently improves single-modality baselines; (2)
our approach outperforms alternative modality fusion methods and strongly
benefits from the proposed chart-based structure; (3) there construction
quality increases with the number of grasps provided; and (4) the touch
information not only enhances the reconstruction at the touch site but also
extrapolates to its local neighborhood.
