  Model-free deep reinforcement learning is sample inefficient. One hypothesis
-- speculated, but not confirmed -- is that catastrophic interference within an
environment inhibits learning. We test this hypothesis through a large-scale
empirical study in the Arcade Learning Environment (ALE) and, indeed, find
supporting evidence. We show that interference causes performance to plateau;
the network cannot train on segments beyond the plateau without degrading the
policy used to reach there. By synthetically controlling for interference, we
demonstrate performance boosts across architectures, learning algorithms and
environments. A more refined analysis shows that learning one segment of a game
often increases prediction errors elsewhere. Our study provides a clear
empirical link between catastrophic interference and sample efficiency in
reinforcement learning.
