  Dexterous manipulation of arbitrary objects, a fundamental daily task for
humans, has been a grand challenge for autonomous robotic systems. Although
data-driven approaches using reinforcement learning can develop specialist
policies that discover behaviors to control a single object, they often exhibit
poor generalization to unseen ones. In this work, we show that policies learned
by existing reinforcement learning algorithms can in fact be generalist when
combined with multi-task learning and a well-chosen object representation. We
show that a single generalist policy can perform in-hand manipulation of over
100 geometrically-diverse real-world objects and generalize to new objects with
unseen shape or size. Interestingly, we find that multi-task learning with
object point cloud representations not only generalizes better but even
outperforms the single-object specialist policies on both training as well as
held-out test objects. Video results at
https://huangwl18.github.io/geometry-dex
