  Environments in Reinforcement Learning are usually only partially observable.
To address this problem, a possible solution is to provide the agent with
information about the past. However, providing complete observations of
numerous steps can be excessive. Inspired by human memory, we propose to
represent history with only important changes in the environment and, in our
approach, to obtain automatically this representation using self-supervision.
Our method (TempAl) aligns temporally-close frames, revealing a general, slowly
varying state of the environment. This procedure is based on contrastive loss,
which pulls embeddings of nearby observations to each other while pushing away
other samples from the batch. It can be interpreted as a metric that captures
the temporal relations of observations. We propose to combine both common
instantaneous and our history representation and we evaluate TempAl on all
available Atari games from the Arcade Learning Environment. TempAl surpasses
the instantaneous-only baseline in 35 environments out of 49. The source code
of the method and of all the experiments is available at
https://github.com/htdt/tempal.
