  Explaining the behaviors of deep neural networks, usually considered as black
boxes, is critical especially when they are now being adopted over diverse
aspects of human life. Taking the advantages of interpretable machine learning
(interpretable ML), this paper proposes a novel tool called Catastrophic
Forgetting Dissector (or CFD) to explain catastrophic forgetting in continual
learning settings. We also introduce a new method called Critical Freezing
based on the observations of our tool. Experiments on ResNet articulate how
catastrophic forgetting happens, particularly showing which components of this
famous network are forgetting. Our new continual learning algorithm defeats
various recent techniques by a significant margin, proving the capability of
the investigation. Critical freezing not only attacks catastrophic forgetting
but also exposes explainability.
