  Handling the ever-increasing scale of contemporary deep learning and
transformer-based models poses a significant challenge. Overparameterized
Transformer networks outperform prior art in Natural Language processing and
Computer Vision. These models contain hundreds of millions of parameters,
demanding significant computational resources and making them prone to
overfitting. In this work we present LayerCollapse, a form of structured
pruning to reduce the depth of fully connected layers. We develop a novel
regularizer allowing for post-training compression without finetuning, while
having limited impact on performance. LayerCollapse controls model
expressiveness with regularization on the activations between fully connected
layers, modulating the linearity of activation functions. A linear activation
function reduces the rank of the transformation to the rank of the
corresponding linear transformation. We demonstrate the effectiveness of
LayerCollapse by showing its compression capabilities in sentimental analysis
and image classification benchmarks. Moreover we show LayerCollapse is an
effective compression aware regularization method in a language modeling
benchmark.
