  With the latest advances in deep learning, there has been a lot of focus on
the online learning paradigm due to its relevance in practical settings.
Although many methods have been investigated for optimal learning settings in
scenarios where the data stream is continuous over time, sparse networks
training in such settings have often been overlooked. In this paper, we explore
the problem of training a neural network with a target sparsity in a particular
case of online learning: the anytime learning at macroscale paradigm (ALMA). We
propose a novel way of progressive pruning, referred to as \textit{Anytime
Progressive Pruning} (APP); the proposed approach significantly outperforms the
baseline dense and Anytime OSP models across multiple architectures and
datasets under short, moderate, and long-sequence training. Our method, for
example, shows an improvement in accuracy of $\approx 7\%$ and a reduction in
the generalization gap by $\approx 22\%$, while being $\approx 1/3$ rd the size
of the dense baseline model in few-shot restricted imagenet training. We
further observe interesting nonmonotonic transitions in the generalization gap
in the high number of megabatches-based ALMA. The code and experiment
dashboards can be accessed at
\url{https://github.com/landskape-ai/Progressive-Pruning} and
\url{https://wandb.ai/landskape/APP}, respectively.
