  Several recent papers have examined generalization in reinforcement learning
(RL), by proposing new environments or ways to add noise to existing
environments, then benchmarking algorithms and model architectures on those
environments. We discuss subtle conceptual properties of RL benchmarks that are
not required in supervised learning (SL), and also properties that an RL
benchmark should possess. Chief among them is one we call the principle of
unchanged optimality: there should exist a single $\pi$ that is optimal across
all train and test tasks. In this work, we argue why this principle is
important, and ways it can be broken or satisfied due to subtle choices in
state representation or model architecture. We conclude by discussing
challenges and future lines of research in theoretically analyzing
generalization benchmarks.
