  Representations are at the core of all deep reinforcement learning (RL)
methods for both Markov decision processes (MDPs) and partially observable
Markov decision processes (POMDPs). Many representation learning methods and
theoretical frameworks have been developed to understand what constitutes an
effective representation. However, the relationships between these methods and
the shared properties among them remain unclear. In this paper, we show that
many of these seemingly distinct methods and frameworks for state and history
abstractions are, in fact, based on a common idea of self-predictive
abstraction. Furthermore, we provide theoretical insights into the widely
adopted objectives and optimization, such as the stop-gradient technique, in
learning self-predictive representations. These findings together yield a
minimalist algorithm to learn self-predictive representations for states and
histories. We validate our theories by applying our algorithm to standard MDPs,
MDPs with distractors, and POMDPs with sparse rewards. These findings culminate
in a set of preliminary guidelines for RL practitioners.
