  Humans are capable of completing a range of challenging manipulation tasks
that require reasoning jointly over modalities such as vision, touch, and
sound. Moreover, many such tasks are partially-observed; for example, taking a
notebook out of a backpack will lead to visual occlusion and require reasoning
over the history of audio or tactile information. While robust tactile sensing
can be costly to capture on robots, microphones near or on a robot's gripper
are a cheap and easy way to acquire audio feedback of contact events, which can
be a surprisingly valuable data source for perception in the absence of vision.
Motivated by the potential for sound to mitigate visual occlusion, we aim to
learn a set of challenging partially-observed manipulation tasks from visual
and audio inputs. Our proposed system learns these tasks by combining offline
imitation learning from a modest number of tele-operated demonstrations and
online finetuning using human provided interventions. In a set of simulated
tasks, we find that our system benefits from using audio, and that by using
online interventions we are able to improve the success rate of offline
imitation learning by ~20%. Finally, we find that our system can complete a set
of challenging, partially-observed tasks on a Franka Emika Panda robot, like
extracting keys from a bag, with a 70% success rate, 50% higher than a policy
that does not use audio.
