  We present a novel predictive model architecture based on the principles of
predictive coding that enables open loop prediction of future observations over
extended horizons. There are two key innovations. First, whereas current
methods typically learn to make long-horizon open-loop predictions using a
multi-step cost function, we instead run the model open loop in the forward
pass during training. Second, current predictive coding models initialize the
representation layer's hidden state to a constant value at the start of an
episode, and consequently typically require multiple steps of interaction with
the environment before the model begins to produce accurate predictions.
Instead, we learn a mapping from the first observation in an episode to the
hidden state, allowing the trained model to immediately produce accurate
predictions. We compare the performance of our architecture to a standard
predictive coding model and demonstrate the ability of the model to make
accurate long horizon open-loop predictions of simulated Doppler radar
altimeter readings during a six degree of freedom Mars landing. Finally, we
demonstrate a 2X reduction in sample complexity by using the model to implement
a Dyna style algorithm to accelerate policy learning with proximal policy
optimization.
