  Generalization remains a challenging problem for deep reinforcement learning
algorithms, which are often trained and tested on the same set of deterministic
game environments. When test environments are unseen and perturbed but the
nature of the task remains the same, generalization gaps can arise. In this
work, we propose and evaluate a surprise minimizing agent on a generalization
benchmark to show an additional reward learned from a simple density model can
show robustness in procedurally generated game environments that provide
constant source of entropy and stochasticity.
