  Catastrophic forgetting and capacity saturation are the central challenges of
any parametric lifelong learning system. In this work, we study these
challenges in the context of sequential supervised learning with an emphasis on
recurrent neural networks. To evaluate the models in the lifelong learning
setting, we propose a curriculum-based, simple, and intuitive benchmark where
the models are trained on tasks with increasing levels of difficulty. To
measure the impact of catastrophic forgetting, the model is tested on all the
previous tasks as it completes any task. As a step towards developing true
lifelong learning systems, we unify Gradient Episodic Memory (a catastrophic
forgetting alleviation approach) and Net2Net(a capacity expansion approach).
Both these models are proposed in the context of feedforward networks and we
evaluate the feasibility of using them for recurrent networks. Evaluation on
the proposed benchmark shows that the unified model is more suitable than the
constituent models for lifelong learning setting.
