  The distributional perspective on reinforcement learning (RL) has given rise
to a series of successful Q-learning algorithms, resulting in state-of-the-art
performance in arcade game environments. However, it has not yet been analyzed
how these findings from a discrete setting translate to complex practical
applications characterized by noisy, high dimensional and continuous
state-action spaces. In this work, we propose Quantile QT-Opt (Q2-Opt), a
distributional variant of the recently introduced distributed Q-learning
algorithm for continuous domains, and examine its behaviour in a series of
simulated and real vision-based robotic grasping tasks. The absence of an actor
in Q2-Opt allows us to directly draw a parallel to the previous discrete
experiments in the literature without the additional complexities induced by an
actor-critic architecture. We demonstrate that Q2-Opt achieves a superior
vision-based object grasping success rate, while also being more sample
efficient. The distributional formulation also allows us to experiment with
various risk distortion metrics that give us an indication of how robots can
concretely manage risk in practice using a Deep RL control policy. As an
additional contribution, we perform batch RL experiments in our virtual
environment and compare them with the latest findings from discrete settings.
Surprisingly, we find that the previous batch RL findings from the literature
obtained on arcade game environments do not generalise to our setup.
