  Dextrous in-hand manipulation with a multi-fingered robotic hand is a
challenging task, esp. when performed with the hand oriented upside down,
demanding permanent force-closure, and when no external sensors are used. For
the task of reorienting an object to a given goal orientation (vs. infinitely
spinning it around an axis), the lack of external sensors is an additional
fundamental challenge as the state of the object has to be estimated all the
time, e.g., to detect when the goal is reached. In this paper, we show that the
task of reorienting a cube to any of the 24 possible goal orientations in a
${\pi}$/2-raster using the torque-controlled DLR-Hand II is possible. The task
is learned in simulation using a modular deep reinforcement learning
architecture: the actual policy has only a small observation time window of
0.5s but gets the cube state as an explicit input which is estimated via a deep
differentiable particle filter trained on data generated by running the policy.
In simulation, we reach a success rate of 92% while applying significant domain
randomization. Via zero-shot Sim2Real-transfer on the real robotic system, all
24 goal orientations can be reached with a high success rate.
