  Dexterous manipulation with a multi-finger hand is one of the most
challenging problems in robotics. While recent progress in imitation learning
has largely improved the sample efficiency compared to Reinforcement Learning,
the learned policy can hardly generalize to manipulate novel objects, given
limited expert demonstrations. In this paper, we propose to learn dexterous
manipulation using large-scale demonstrations with diverse 3D objects in a
category, which are generated from a human grasp affordance model. This
generalizes the policy to novel object instances within the same category. To
train the policy, we propose a novel imitation learning objective jointly with
a geometric representation learning objective using our demonstrations. By
experimenting with relocating diverse objects in simulation, we show that our
approach outperforms baselines with a large margin when manipulating novel
objects. We also ablate the importance on 3D object representation learning for
manipulation. We include videos, code, and additional information on the
project website - https://kristery.github.io/ILAD/ .
