  Continual learning on edge devices poses unique challenges due to stringent
resource constraints. This paper introduces a novel method that leverages
stochastic competition principles to promote sparsity, significantly reducing
deep network memory footprint and computational demand. Specifically, we
propose deep networks that comprise blocks of units that compete locally to win
the representation of each arising new task; competition takes place in a
stochastic manner. This type of network organization results in sparse
task-specific representations from each network layer; the sparsity pattern is
obtained during training and is different among tasks. Crucially, our method
sparsifies both the weights and the weight gradients, thus facilitating
training on edge devices. This is performed on the grounds of winning
probability for each unit in a block. During inference, the network retains
only the winning unit and zeroes-out all weights pertaining to non-winning
units for the task at hand. Thus, our approach is specifically tailored for
deployment on edge devices, providing an efficient and scalable solution for
continual learning in resource-limited environments.
