  The rise of Deep Neural Networks (DNNs) has led to an increase in model size
and complexity, straining the memory capacity of GPUs. Sparsity in DNNs,
characterized as structural or ephemeral, has gained attention as a solution.
This work focuses on ephemeral sparsity, aiming to reduce memory consumption
during training. It emphasizes the significance of activations, an often
overlooked component, and their role in memory usage. This work employs
structured pruning in Block Sparse Compressed Row (BSR) format in combination
with a magnitude-based criterion to efficiently prune activations. We
furthermore introduce efficient block-sparse operators for GPUs and showcase
their effectiveness, as well as the superior compression offered by block
sparsity. We report the effectiveness of activation pruning by evaluating
training speed, accuracy, and memory usage of large-scale neural architectures
on the example of ResMLP on image classification tasks. As a result, we observe
a memory reduction of up to 32% while maintaining accuracy. Ultimately, our
approach aims to democratize large-scale model training, reduce GPU
requirements, and address ecological concerns.
