  Deep representation learning methods struggle with continual learning,
suffering from both catastrophic forgetting of useful units and loss of
plasticity, often due to rigid and unuseful units. While many methods address
these two issues separately, only a few currently deal with both
simultaneously. In this paper, we introduce Utility-based Perturbed Gradient
Descent (UPGD) as a novel approach for the continual learning of
representations. UPGD combines gradient updates with perturbations, where it
applies smaller modifications to more useful units, protecting them from
forgetting, and larger modifications to less useful units, rejuvenating their
plasticity. We use a challenging streaming learning setup where continual
learning problems have hundreds of non-stationarities and unknown task
boundaries. We show that many existing methods suffer from at least one of the
issues, predominantly manifested by their decreasing accuracy over tasks. On
the other hand, UPGD continues to improve performance and surpasses or is
competitive with all methods in all problems. Finally, in extended
reinforcement learning experiments with PPO, we show that while Adam exhibits a
performance drop after initial learning, UPGD avoids it by addressing both
continual learning issues.
