  Deep reinforcement learning (RL) has achieved breakthrough results on many
tasks, but agents often fail to generalize beyond the environment they were
trained in. As a result, deep RL algorithms that promote generalization are
receiving increasing attention. However, works in this area use a wide variety
of tasks and experimental setups for evaluation. The literature lacks a
controlled assessment of the merits of different generalization schemes. Our
aim is to catalyze community-wide progress on generalization in deep RL. To
this end, we present a benchmark and experimental protocol, and conduct a
systematic empirical study. Our framework contains a diverse set of
environments, our methodology covers both in-distribution and
out-of-distribution generalization, and our evaluation includes deep RL
algorithms that specifically tackle generalization. Our key finding is that
`vanilla' deep RL algorithms generalize better than specialized schemes that
were proposed specifically to tackle generalization.
