  We propose a sim-to-real framework for dexterous manipulation which can
generalize to new objects of the same category in the real world. The key of
our framework is to train the manipulation policy with point cloud inputs and
dexterous hands. We propose two new techniques to enable joint learning on
multiple objects and sim-to-real generalization: (i) using imagined hand point
clouds as augmented inputs; and (ii) designing novel contact-based rewards. We
empirically evaluate our method using an Allegro Hand to grasp novel objects in
both simulation and real world. To the best of our knowledge, this is the first
policy learning-based framework that achieves such generalization results with
dexterous hands. Our project page is available at
https://yzqin.github.io/dexpoint
