  Despite the significant progress of deep reinforcement learning (RL) in
solving sequential decision making problems, RL agents often overfit to
training environments and struggle to adapt to new, unseen environments. This
prevents robust applications of RL in real world situations, where system
dynamics may deviate wildly from the training settings. In this work, our
primary contribution is to propose an information theoretic regularization
objective and an annealing-based optimization method to achieve better
generalization ability in RL agents. We demonstrate the extreme generalization
benefits of our approach in different domains ranging from maze navigation to
robotic tasks; for the first time, we show that agents can generalize to test
parameters more than 10 standard deviations away from the training parameter
distribution. This work provides a principled way to improve generalization in
RL by gradually removing information that is redundant for task-solving; it
opens doors for the systematic study of generalization from training to
extremely different testing settings, focusing on the established connections
between information theory and machine learning.
