  Continual lifelong learning is essential to many applications. In this paper,
we propose a simple but effective approach to continual deep learning. Our
approach leverages the principles of deep model compression, critical weights
selection, and progressive networks expansion. By enforcing their integration
in an iterative manner, we introduce an incremental learning method that is
scalable to the number of sequential tasks in a continual learning process. Our
approach is easy to implement and owns several favorable characteristics.
First, it can avoid forgetting (i.e., learn new tasks while remembering all
previous tasks). Second, it allows model expansion but can maintain the model
compactness when handling sequential tasks. Besides, through our compaction and
selection/expansion mechanism, we show that the knowledge accumulated through
learning previous tasks is helpful to build a better model for the new tasks
compared to training the models independently with tasks. Experimental results
show that our approach can incrementally learn a deep model tackling multiple
tasks without forgetting, while the model compactness is maintained with the
performance more satisfiable than individual task training.
