  Modern representation learning methods often struggle to adapt quickly under
non-stationarity because they suffer from catastrophic forgetting and decaying
plasticity. Such problems prevent learners from fast adaptation since they may
forget useful features or have difficulty learning new ones. Hence, these
methods are rendered ineffective for continual learning. This paper proposes
Utility-based Perturbed Gradient Descent (UPGD), an online learning algorithm
well-suited for continual learning agents. UPGD protects useful weights or
features from forgetting and perturbs less useful ones based on their
utilities. Our empirical results show that UPGD helps reduce forgetting and
maintain plasticity, enabling modern representation learning methods to work
effectively in continual learning.
