  We present a data-efficient framework for solving visuomotor sequential
decision-making problems which exploits the combination of reinforcement
learning (RL) and latent variable generative models. Our framework trains deep
visuomotor policies by introducing an action latent variable such that the
feed-forward policy search can be divided into three parts: (i) training a
sub-policy that outputs a distribution over the action latent variable given a
state of the system, (ii) unsupervised training of a generative model that
outputs a sequence of motor actions conditioned on the latent action variable,
and (iii) supervised training of the deep visuomotor policy in an end-to-end
fashion. Our approach enables safe exploration and alleviates the
data-inefficiency problem as it exploits prior knowledge about valid sequences
of motor actions. Moreover, we provide a set of measures for evaluation of
generative models such that we are able to predict the performance of the RL
policy training prior to the actual training on a physical robot. We define two
novel measures of disentanglement and local linearity for assessing the quality
of latent representations, and complement them with existing measures for
assessment of the learned distribution. We experimentally determine the
characteristics of different generative models that have the most influence on
performance of the final policy training on a robotic picking task.
