  We present a policy search method for learning complex feedback control
policies that map from high-dimensional sensory inputs to motor torques, for
manipulation tasks with discontinuous contact dynamics. We build on a prior
technique called guided policy search (GPS), which iteratively optimizes a set
of local policies for specific instances of a task, and uses these to train a
complex, high-dimensional global policy that generalizes across task instances.
We extend GPS in the following ways: (1) we propose the use of a model-free
local optimizer based on path integral stochastic optimal control (PI2), which
enables us to learn local policies for tasks with highly discontinuous contact
dynamics; and (2) we enable GPS to train on a new set of task instances in
every iteration by using on-policy sampling: this increases the diversity of
the instances that the policy is trained on, and is crucial for achieving good
generalization. We show that these contributions enable us to learn deep neural
network policies that can directly perform torque control from visual input. We
validate the method on a challenging door opening task and a pick-and-place
task, and we demonstrate that our approach substantially outperforms the prior
LQR-based local policy optimizer on these tasks. Furthermore, we show that
on-policy sampling significantly increases the generalization ability of these
policies.
