  Representation learning and exploration are among the key challenges for any
deep reinforcement learning agent. In this work, we provide a singular value
decomposition based method that can be used to obtain representations that
preserve the underlying transition structure in the domain. Perhaps
interestingly, we show that these representations also capture the relative
frequency of state visitations, thereby providing an estimate for pseudo-counts
for free. To scale this decomposition method to large-scale domains, we provide
an algorithm that never requires building the transition matrix, can make use
of deep networks, and also permits mini-batch training. Further, we draw
inspiration from predictive state representations and extend our decomposition
method to partially observable environments. With experiments on multi-task
settings with partially observable domains, we show that the proposed method
can not only learn useful representation on DM-Lab-30 environments (that have
inputs involving language instructions, pixel images, and rewards, among
others) but it can also be effective at hard exploration tasks in DM-Hard-8
environments.
