  Modern reinforcement learning algorithms reach super-human performance on
many board and video games, but they are sample inefficient, i.e. they
typically require significantly more playing experience than humans to reach an
equal performance level. To improve sample efficiency, an agent may build a
model of the environment and use planning methods to update its policy. In this
article we introduce Variational State Tabulation (VaST), which maps an
environment with a high-dimensional state space (e.g. the space of visual
inputs) to an abstract tabular model. Prioritized sweeping with small backups,
a highly efficient planning method, can then be used to update state-action
values. We show how VaST can rapidly learn to maximize reward in tasks like 3D
navigation and efficiently adapt to sudden changes in rewards or transition
probabilities.
