  Recent advances in deep reinforcement learning have made significant strides
in performance on applications such as Go and Atari games. However, developing
practical methods to balance exploration and exploitation in complex domains
remains largely unsolved. Thompson Sampling and its extension to reinforcement
learning provide an elegant approach to exploration that only requires access
to posterior samples of the model. At the same time, advances in approximate
Bayesian methods have made posterior approximation for flexible neural network
models practical. Thus, it is attractive to consider approximate Bayesian
neural networks in a Thompson Sampling framework. To understand the impact of
using an approximate posterior on Thompson Sampling, we benchmark
well-established and recently developed methods for approximate posterior
sampling combined with Thompson Sampling over a series of contextual bandit
problems. We found that many approaches that have been successful in the
supervised learning setting underperformed in the sequential decision-making
scenario. In particular, we highlight the challenge of adapting slowly
converging uncertainty estimates to the online setting.
