  Unsupervised exploration and representation learning become increasingly
important when learning in diverse and sparse environments. The
information-theoretic principle of empowerment formalizes an unsupervised
exploration objective through an agent trying to maximize its influence on the
future states of its environment. Previous approaches carry certain limitations
in that they either do not employ closed-loop feedback or do not have an
internal state. As a consequence, a privileged final state is taken as an
influence measure, rather than the full trajectory. We provide a model-free
method which takes into account the whole trajectory while still offering the
benefits of option-based approaches. We successfully apply our approach to
settings with large action spaces, where discovery of meaningful action
sequences is particularly difficult.
