  We introduce Recurrent Predictive State Policy (RPSP) networks, a recurrent
architecture that brings insights from predictive state representations to
reinforcement learning in partially observable environments. Predictive state
policy networks consist of a recursive filter, which keeps track of a belief
about the state of the environment, and a reactive policy that directly maps
beliefs to actions, to maximize the cumulative reward. The recursive filter
leverages predictive state representations (PSRs) (Rosencrantz and Gordon,
2004; Sun et al., 2016) by modeling predictive state-- a prediction of the
distribution of future observations conditioned on history and future actions.
This representation gives rise to a rich class of statistically consistent
algorithms (Hefny et al., 2018) to initialize the recursive filter. Predictive
state serves as an equivalent representation of a belief state. Therefore, the
policy component of the RPSP-network can be purely reactive, simplifying
training while still allowing optimal behaviour. Moreover, we use the PSR
interpretation during training as well, by incorporating prediction error in
the loss function. The entire network (recursive filter and reactive policy) is
still differentiable and can be trained using gradient based methods. We
optimize our policy using a combination of policy gradient based on rewards
(Williams, 1992) and gradient descent based on prediction error. We show the
efficacy of RPSP-networks under partial observability on a set of robotic
control tasks from OpenAI Gym. We empirically show that RPSP-networks perform
well compared with memory-preserving networks such as GRUs, as well as finite
memory models, being the overall best performing method.
