  Learning a good history representation is one of the core challenges of
reinforcement learning (RL) in partially observable environments. Recent works
have shown the advantages of various auxiliary tasks for facilitating
representation learning. However, the effectiveness of such auxiliary tasks has
not been fully convincing, especially in partially observable environments that
require long-term memorization and inference. In this empirical study, we
investigate the effectiveness of future prediction for learning the
representations of histories, possibly of extensive length, in partially
observable environments. We first introduce an approach that decouples the task
of learning history representations from policy optimization via future
prediction. Then, our main contributions are two-fold: (a) we demonstrate that
the performance of reinforcement learning is strongly correlated with the
prediction accuracy of future observations in partially observable
environments, and (b) our approach can significantly improve the overall
end-to-end approach by preventing high-variance noisy signals from
reinforcement learning objectives to influence the representation learning. We
illustrate our claims on three types of benchmarks that necessitate the ability
to process long histories for high returns.
