  While deep learning has led to remarkable advances across diverse
applications, it struggles in domains where the data distribution changes over
the course of learning. In stark contrast, biological neural networks
continually adapt to changing domains, possibly by leveraging complex molecular
machinery to solve many tasks simultaneously. In this study, we introduce
intelligent synapses that bring some of this biological complexity into
artificial neural networks. Each synapse accumulates task relevant information
over time, and exploits this information to rapidly store new memories without
forgetting old ones. We evaluate our approach on continual learning of
classification tasks, and show that it dramatically reduces forgetting while
maintaining computational efficiency.
