  Unsupervised representation learning has succeeded with excellent results in
many applications. It is an especially powerful tool to learn a good
representation of environments with partial or noisy observations. In partially
observable domains it is important for the representation to encode a belief
state, a sufficient statistic of the observations seen so far. In this paper,
we investigate whether it is possible to learn such a belief representation
using modern neural architectures. Specifically, we focus on one-step frame
prediction and two variants of contrastive predictive coding (CPC) as the
objective functions to learn the representations. To evaluate these learned
representations, we test how well they can predict various pieces of
information about the underlying state of the environment, e.g., position of
the agent in a 3D maze. We show that all three methods are able to learn belief
representations of the environment, they encode not only the state information,
but also its uncertainty, a crucial aspect of belief states. We also find that
for CPC multi-step predictions and action-conditioning are critical for
accurate belief representations in visually complex environments. The ability
of neural representations to capture the belief information has the potential
to spur new advances for learning and planning in partially observable domains,
where leveraging uncertainty is essential for optimal decision making.
