  In this paper, we propose a continual learning (CL) technique that is
beneficial to sequential task learners by improving their retained accuracy and
reducing catastrophic forgetting. The principal target of our approach is the
automatic extraction of modular parts of the neural network and then estimating
the relatedness between the tasks given these modular components. This
technique is applicable to different families of CL methods such as
regularization-based (e.g., the Elastic Weight Consolidation) or the
rehearsal-based (e.g., the Gradient Episodic Memory) approaches where episodic
memory is needed. Empirical results demonstrate remarkable performance gain (in
terms of robustness to forgetting) for methods such as EWC and GEM based on our
technique, especially when the memory budget is very limited.
