  Reinforcement learning algorithms typically rely on the assumption that the
environment dynamics and value function can be expressed in terms of a
Markovian state representation. However, when state information is only
partially observable, how can an agent learn such a state representation, and
how can it detect when it has found one? We introduce a metric that can
accomplish both objectives, without requiring access to--or knowledge of--an
underlying, unobservable state space. Our metric, the $\lambda$-discrepancy, is
the difference between two distinct temporal difference (TD) value estimates,
each computed using TD($\lambda$) with a different value of $\lambda$. Since
TD($\lambda$=0) makes an implicit Markov assumption and TD($\lambda$=1) does
not, a discrepancy between these estimates is a potential indicator of a
non-Markovian state representation. Indeed, we prove that the
$\lambda$-discrepancy is exactly zero for all Markov decision processes and
almost always non-zero for a broad class of partially observable environments.
We also demonstrate empirically that, once detected, minimizing the
$\lambda$-discrepancy can help with learning a memory function to mitigate the
corresponding partial observability. We then train a reinforcement learning
agent that simultaneously constructs two recurrent value networks with
different $\lambda$ parameters and minimizes the difference between them as an
auxiliary loss. The approach scales to challenging partially observable
domains, where the resulting agent frequently performs significantly better
(and never performs worse) than a baseline recurrent agent with only a single
value network.
