  Efficient exploration remains a major challenge for reinforcement learning.
One reason is that the variability of the returns often depends on the current
state and action, and is therefore heteroscedastic. Classical exploration
strategies such as upper confidence bound algorithms and Thompson sampling fail
to appropriately account for heteroscedasticity, even in the bandit setting.
Motivated by recent findings that address this issue in bandits, we propose to
use Information-Directed Sampling (IDS) for exploration in reinforcement
learning. As our main contribution, we build on recent advances in
distributional reinforcement learning and propose a novel, tractable
approximation of IDS for deep Q-learning. The resulting exploration strategy
explicitly accounts for both parametric uncertainty and heteroscedastic
observation noise. We evaluate our method on Atari games and demonstrate a
significant improvement over alternative approaches.
