  This paper introduces a method for constructing an upper bound for
exploration policy using either the weighted variance of return sequences or
the weighted temporal difference (TD) error. We demonstrate that the variance
of the return sequence for a specific state-action pair is an important
information source that can be leveraged to guide exploration in reinforcement
learning. The intuition is that fluctuation in the return sequence indicates
greater uncertainty in the near future returns. This divergence occurs because
of the cyclic nature of value-based reinforcement learning; the evolving value
function begets policy improvements which in turn modify the value function.
Although both variance and TD errors capture different aspects of this
uncertainty, our analysis shows that both can be valuable to guide exploration.
We propose a two-stream network architecture to estimate weighted variance/TD
errors within DQN agents for our exploration method and show that it
outperforms the baseline on a wide range of Atari games.
