  Deep Neural Networks (DNNs) have achieved remarkable success in addressing
many previously unsolvable tasks. However, the storage and computational
requirements associated with DNNs pose a challenge for deploying these trained
models on resource-limited devices. Therefore, a plethora of compression and
pruning techniques have been proposed in recent years. Low-rank decomposition
techniques are among the approaches most utilized to address this problem.
Compared to post-training compression, compression-promoted training is still
under-explored. In this paper, we present a theoretically-justified novel
approach, termed Low-Rank Induced Training (LoRITa), that promotes low-rankness
through the composition of linear layers and compresses by using singular value
truncation. This is achieved without the need to change the structure at
inference time or require constrained and/or additional optimization, other
than the standard weight decay regularization. Moreover, LoRITa eliminates the
need to (i) initialize with pre-trained models and (ii) specify rank selection
prior to training. Our experimental results (i) demonstrate the effectiveness
of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision
Transformers, and CIFAR10/100 on Convolutional Neural Networks, and (ii)
illustrate that we achieve either competitive or SOTA results when compared to
leading structured pruning methods in terms of FLOPs and parameters drop.
