  A fundamental challenge in teaching robots is to provide an effective
interface for human teachers to demonstrate useful skills to a robot. This
challenge is exacerbated in dexterous manipulation, where teaching
high-dimensional, contact-rich behaviors often require esoteric teleoperation
tools. In this work, we present Holo-Dex, a framework for dexterous
manipulation that places a teacher in an immersive mixed reality through
commodity VR headsets. The high-fidelity hand pose estimator onboard the
headset is used to teleoperate the robot and collect demonstrations for a
variety of general-purpose dexterous tasks. Given these demonstrations, we use
powerful feature learning combined with non-parametric imitation to train
dexterous skills. Our experiments on six common dexterous tasks, including
in-hand rotation, spinning, and bottle opening, indicate that Holo-Dex can both
collect high-quality demonstration data and train skills in a matter of hours.
Finally, we find that our trained skills can exhibit generalization on objects
not seen in training. Videos of Holo-Dex are available at
https://holo-dex.github.io.
