  Sleep plays an important role in incremental learning and consolidation of
memories in biological systems. Motivated by the processes that are known to be
involved in sleep generation in biological networks, we developed an algorithm
that implements a sleep-like phase in artificial neural networks (ANNs). After
initial training phase, we convert the ANN to a spiking neural network (SNN)
and simulate an offline sleep-like phase using spike-timing dependent
plasticity rules to modify synaptic weights. The SNN is then converted back to
the ANN and evaluated or trained on new inputs. We demonstrate several
performance improvements after applying this processing to ANNs trained on
MNIST, CUB200 and a motivating toy dataset. First, in an incremental learning
framework, sleep is able to recover older tasks that were otherwise forgotten
in the ANN without sleep phase due to catastrophic forgetting. Second, sleep
results in forward transfer learning of unseen tasks. Finally, sleep improves
generalization ability of the ANNs to classify images with various types of
noise. We provide a theoretical basis for the beneficial role of the
brain-inspired sleep-like phase for the ANNs and present an algorithmic way for
future implementations of the various features of sleep in deep learning ANNs.
Overall, these results suggest that biological sleep can help mitigate a number
of problems ANNs suffer from, such as poor generalization and catastrophic
forgetting for incremental learning.
