  Autonomous agents trained using deep reinforcement learning (RL) often lack
the ability to successfully generalise to new environments, even when these
environments share characteristics with the ones they have encountered during
training. In this work, we investigate how the sampling of individual
environment instances, or levels, affects the zero-shot generalisation (ZSG)
ability of RL agents. We discover that, for deep actor-critic architectures
sharing their base layers, prioritising levels according to their value loss
minimises the mutual information between the agent's internal representation
and the set of training levels in the generated training data. This provides a
novel theoretical justification for the regularisation achieved by certain
adaptive sampling strategies. We then turn our attention to unsupervised
environment design (UED) methods, which assume control over level generation.
We find that existing UED methods can significantly shift the training
distribution, which translates to low ZSG performance. To prevent both
overfitting and distributional shift, we introduce data-regularised environment
design (DRED). DRED generates levels using a generative model trained to
approximate the ground truth distribution of an initial set of level
parameters. Through its grounding, DRED achieves significant improvements in
ZSG over adaptive level sampling strategies and UED methods. Our code and
experimental data are available at https://github.com/uoe-agents/dred.
