  Mutual information maximization provides an appealing formalism for learning
representations of data. In the context of reinforcement learning (RL), such
representations can accelerate learning by discarding irrelevant and redundant
information, while retaining the information necessary for control. Much of the
prior work on these methods has addressed the practical difficulties of
estimating mutual information from samples of high-dimensional observations,
while comparatively less is understood about which mutual information
objectives yield representations that are sufficient for RL from a theoretical
perspective. In this paper, we formalize the sufficiency of a state
representation for learning and representing the optimal policy, and study
several popular mutual-information based objectives through this lens.
Surprisingly, we find that two of these objectives can yield insufficient
representations given mild and common assumptions on the structure of the MDP.
We corroborate our theoretical results with empirical experiments on a
simulated game environment with visual observations.
