  Deep reinforcement learning (RL) agents trained in a limited set of
environments tend to suffer overfitting and fail to generalize to unseen
testing environments. To improve their generalizability, data augmentation
approaches (e.g. cutout and random convolution) are previously explored to
increase the data diversity. However, we find these approaches only locally
perturb the observations regardless of the training environments, showing
limited effectiveness on enhancing the data diversity and the generalization
performance. In this work, we introduce a simple approach, named mixreg, which
trains agents on a mixture of observations from different training environments
and imposes linearity constraints on the observation interpolations and the
supervision (e.g. associated reward) interpolations. Mixreg increases the data
diversity more effectively and helps learn smoother policies. We verify its
effectiveness on improving generalization by conducting extensive experiments
on the large-scale Procgen benchmark. Results show mixreg outperforms the
well-established baselines on unseen testing environments by a large margin.
Mixreg is simple, effective and general. It can be applied to both policy-based
and value-based RL algorithms. Code is available at
https://github.com/kaixin96/mixreg .
