  Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in
recent years. However, these models have been getting increasingly large as
they become more accurate and safe. This means that their training becomes
increasingly costly and time-consuming and typically yields a single model to
fit all targets. Various techniques have been proposed in the literature to
mitigate this, including pruning, sparsification, or quantization of model
weights and updates. While achieving high compression rates, they often incur
significant computational overheads at training or lead to non-negligible
accuracy penalty. Alternatively, factorization methods have been leveraged for
low-rank compression of DNNs. Similarly, such techniques (e.g., SVD) frequently
rely on heavy iterative decompositions of layers and are potentially
sub-optimal for non-linear models, such as DNNs. We take a further step in
designing efficient low-rank models and propose Maestro, a framework for
trainable low-rank layers. Instead of iteratively applying a priori
decompositions, the low-rank structure is baked into the training process
through LoD, a low-rank ordered decomposition. Not only is this the first time
importance ordering via sampling is applied on the decomposed DNN structure,
but it also allows selecting ranks at a layer granularity. Our theoretical
analysis demonstrates that in special cases LoD recovers the SVD decomposition
and PCA. Applied to DNNs, Maestro enables the extraction of lower footprint
models that preserve performance. Simultaneously, it enables the graceful
trade-off between accuracy-latency for deployment to even more constrained
devices without retraining.
