  One of the key challenges in visual Reinforcement Learning (RL) is to learn
policies that can generalize to unseen environments. Recently, data
augmentation techniques aiming at enhancing data diversity have demonstrated
proven performance in improving the generalization ability of learned policies.
However, due to the sensitivity of RL training, naively applying data
augmentation, which transforms each pixel in a task-agnostic manner, may suffer
from instability and damage the sample efficiency, thus further exacerbating
the generalization performance. At the heart of this phenomenon is the diverged
action distribution and high-variance value estimation in the face of augmented
images. To alleviate this issue, we propose Task-aware Lipschitz Data
Augmentation (TLDA) for visual RL, which explicitly identifies the
task-correlated pixels with large Lipschitz constants, and only augments the
task-irrelevant pixels. To verify the effectiveness of TLDA, we conduct
extensive experiments on DeepMind Control suite, CARLA and DeepMind
Manipulation tasks, showing that TLDA improves both sample efficiency in
training time and generalization in test time. It outperforms previous
state-of-the-art methods across the 3 different visual control benchmarks.
