  Guided policy search is a method for reinforcement learning that trains a
general policy for accomplishing a given task by guiding the learning of the
policy with multiple guiding distributions. Guided policy search relies on
learning an underlying dynamical model of the environment and then, at each
iteration of the algorithm, using that model to gradually improve the policy.
This model, though, often makes the assumption that the environment dynamics
are markovian, e.g., depend only on the current state and control signal. In
this paper we apply guided policy search to a problem with non-markovian
dynamics. Specifically, we apply it to the problem of pouring a precise amount
of liquid from a cup into a bowl, where many of the sensor measurements
experience non-trivial amounts of delay. We show that, with relatively simple
state augmentation, guided policy search can be extended to non-markovian
dynamical systems, where the non-markovianess is caused by delayed sensor
readings.
