  Exploration is a fundamental aspect of Reinforcement Learning, typically
implemented using stochastic action-selection. Exploration, however, can be
more efficient if directed toward gaining new world knowledge. Visit-counters
have been proven useful both in practice and in theory for directed
exploration. However, a major limitation of counters is their locality. While
there are a few model-based solutions to this shortcoming, a model-free
approach is still missing. We propose $E$-values, a generalization of counters
that can be used to evaluate the propagating exploratory value over
state-action trajectories. We compare our approach to commonly used RL
techniques, and show that using $E$-values improves learning and performance
over traditional counters. We also show how our method can be implemented with
function approximation to efficiently learn continuous MDPs. We demonstrate
this by showing that our approach surpasses state of the art performance in the
Freeway Atari 2600 game.
