  Aiming to replicate human-like dexterity, perceptual experiences, and motion
patterns, we explore learning from human demonstrations using a bimanual system
with multifingered hands and visuotactile data. Two significant challenges
exist: the lack of an affordable and accessible teleoperation system suitable
for a dual-arm setup with multifingered hands, and the scarcity of
multifingered hand hardware equipped with touch sensing. To tackle the first
challenge, we develop HATO, a low-cost hands-arms teleoperation system that
leverages off-the-shelf electronics, complemented with a software suite that
enables efficient data collection; the comprehensive software suite also
supports multimodal data processing, scalable policy learning, and smooth
policy deployment. To tackle the latter challenge, we introduce a novel
hardware adaptation by repurposing two prosthetic hands equipped with touch
sensors for research. Using visuotactile data collected from our system, we
learn skills to complete long-horizon, high-precision tasks which are difficult
to achieve without multifingered dexterity and touch feedback. Furthermore, we
empirically investigate the effects of dataset size, sensing modality, and
visual input preprocessing on policy learning. Our results mark a promising
step forward in bimanual multifingered manipulation from visuotactile data.
Videos, code, and datasets can be found at https://toruowo.github.io/hato/ .
