  In-hand object reorientation is necessary for performing many dexterous
manipulation tasks, such as tool use in less structured environments that
remain beyond the reach of current robots. Prior works built reorientation
systems assuming one or many of the following: reorienting only specific
objects with simple shapes, limited range of reorientation, slow or quasistatic
manipulation, simulation-only results, the need for specialized and costly
sensor suites, and other constraints which make the system infeasible for
real-world deployment. We present a general object reorientation controller
that does not make these assumptions. It uses readings from a single commodity
depth camera to dynamically reorient complex and new object shapes by any
rotation in real-time, with the median reorientation time being close to seven
seconds. The controller is trained using reinforcement learning in simulation
and evaluated in the real world on new object shapes not used for training,
including the most challenging scenario of reorienting objects held in the air
by a downward-facing hand that must counteract gravity during reorientation.
Our hardware platform only uses open-source components that cost less than five
thousand dollars. Although we demonstrate the ability to overcome assumptions
in prior work, there is ample scope for improving absolute performance. For
instance, the challenging duck-shaped object not used for training was dropped
in 56 percent of the trials. When it was not dropped, our controller reoriented
the object within 0.4 radians (23 degrees) 75 percent of the time. Videos are
available at: https://taochenshh.github.io/projects/visual-dexterity.
