  Self-supervised learning and data augmentation have significantly reduced the
performance gap between state and image-based reinforcement learning agents in
continuous control tasks. However, it is still unclear whether current
techniques can face a variety of visual conditions required by real-world
environments. We propose a challenging benchmark that tests agents' visual
generalization by adding graphical variety to existing continuous control
domains. Our empirical analysis shows that current methods struggle to
generalize across a diverse set of visual changes, and we examine the specific
factors of variation that make these tasks difficult. We find that data
augmentation techniques outperform self-supervised learning approaches and that
more significant image transformations provide better visual generalization
\footnote{The benchmark and our augmented actor-critic implementation are
open-sourced @ https://github.com/QData/dmc_remastered)
