  Forgetting is often seen as an unwanted characteristic in both human and
machine learning. However, we propose that forgetting can in fact be favorable
to learning. We introduce "forget-and-relearn" as a powerful paradigm for
shaping the learning trajectories of artificial neural networks. In this
process, the forgetting step selectively removes undesirable information from
the model, and the relearning step reinforces features that are consistently
useful under different conditions. The forget-and-relearn framework unifies
many existing iterative training algorithms in the image classification and
language emergence literature, and allows us to understand the success of these
algorithms in terms of the disproportionate forgetting of undesirable
information. We leverage this understanding to improve upon existing algorithms
by designing more targeted forgetting operations. Insights from our analysis
provide a coherent view on the dynamics of iterative training in neural
networks and offer a clear path towards performance improvements.
