  In this paper, we focus on online representation learning in non-stationary
environments which may require continuous adaptation of model architecture. We
propose a novel online dictionary-learning (sparse-coding) framework which
incorporates the addition and deletion of hidden units (dictionary elements),
and is inspired by the adult neurogenesis phenomenon in the dentate gyrus of
the hippocampus, known to be associated with improved cognitive function and
adaptation to new environments. In the online learning setting, where new input
instances arrive sequentially in batches, the neuronal-birth is implemented by
adding new units with random initial weights (random dictionary elements); the
number of new units is determined by the current performance (representation
error) of the dictionary, higher error causing an increase in the birth rate.
Neuronal-death is implemented by imposing l1/l2-regularization (group sparsity)
on the dictionary within the block-coordinate descent optimization at each
iteration of our online alternating minimization scheme, which iterates between
the code and dictionary updates. Finally, hidden unit connectivity adaptation
is facilitated by introducing sparsity in dictionary elements. Our empirical
evaluation on several real-life datasets (images and language) as well as on
synthetic data demonstrates that the proposed approach can considerably
outperform the state-of-art fixed-size (nonadaptive) online sparse coding of
Mairal et al. (2009) in the presence of nonstationary data. Moreover, we
identify certain properties of the data (e.g., sparse inputs with nearly
non-overlapping supports) and of the model (e.g., dictionary sparsity)
associated with such improvements.
