  We show that a purely tactile dextrous in-hand manipulation task with
continuous regrasping, requiring permanent force closure, can be learned from
scratch and executed robustly on a torque-controlled humanoid robotic hand. The
task is rotating a cube without dropping it, but in contrast to OpenAI's
seminal cube manipulation task, the palm faces downwards and no cameras but
only the hand's position and torque sensing are used. Although the task seems
simple, it combines for the first time all the challenges in execution as well
as learning that are important for using in-hand manipulation in real-world
applications. We efficiently train in a precisely modeled and identified rigid
body simulation with off-policy deep reinforcement learning, significantly sped
up by a domain adapted curriculum, leading to a moderate 600 CPU hours of
training time. The resulting policy is robustly transferred to the real
humanoid DLR Hand-II, e.g., reaching more than 46 full 2${\pi}$ rotations of
the cube in a single run and allowing for disturbances like different cube
sizes, hand orientation, or pulling a finger.
