  Visual Reinforcement Learning (Visual RL), coupled with high-dimensional
observations, has consistently confronted the long-standing challenge of
out-of-distribution generalization. Despite the focus on algorithms aimed at
resolving visual generalization problems, we argue that the devil is in the
existing benchmarks as they are restricted to isolated tasks and generalization
categories, undermining a comprehensive evaluation of agents' visual
generalization capabilities. To bridge this gap, we introduce RL-ViGen: a novel
Reinforcement Learning Benchmark for Visual Generalization, which contains
diverse tasks and a wide spectrum of generalization types, thereby facilitating
the derivation of more reliable conclusions. Furthermore, RL-ViGen incorporates
the latest generalization visual RL algorithms into a unified framework, under
which the experiment results indicate that no single existing algorithm has
prevailed universally across tasks. Our aspiration is that RL-ViGen will serve
as a catalyst in this area, and lay a foundation for the future creation of
universal visual generalization RL agents suitable for real-world scenarios.
Access to our code and implemented algorithms is provided at
https://gemcollector.github.io/RL-ViGen/.
