  Despite remarkable successes, deep reinforcement learning algorithms remain
sample inefficient: they require an enormous amount of trial and error to find
good policies. Model-based algorithms promise sample efficiency by building an
environment model that can be used for planning. Posterior Sampling for
Reinforcement Learning is such a model-based algorithm that has attracted
significant interest due to its performance in the tabular setting. This paper
introduces Posterior Sampling for Deep Reinforcement Learning (PSDRL), the
first truly scalable approximation of Posterior Sampling for Reinforcement
Learning that retains its model-based essence. PSDRL combines efficient
uncertainty quantification over latent state space models with a specially
tailored continual planning algorithm based on value-function approximation.
Extensive experiments on the Atari benchmark show that PSDRL significantly
outperforms previous state-of-the-art attempts at scaling up posterior sampling
while being competitive with a state-of-the-art (model-based) reinforcement
learning method, both in sample efficiency and computational efficiency.
