  Factorized layers--operations parameterized by products of two or more
matrices--occur in a variety of deep learning contexts, including compressed
model training, certain types of knowledge distillation, and multi-head
self-attention architectures. We study how to initialize and regularize deep
nets containing such layers, examining two simple, understudied schemes,
spectral initialization and Frobenius decay, for improving their performance.
The guiding insight is to design optimization routines for these networks that
are as close as possible to that of their well-tuned, non-decomposed
counterparts; we back this intuition with an analysis of how the initialization
and regularization schemes impact training with gradient descent, drawing on
modern attempts to understand the interplay of weight-decay and
batch-normalization. Empirically, we highlight the benefits of spectral
initialization and Frobenius decay across a variety of settings. In model
compression, we show that they enable low-rank methods to significantly
outperform both unstructured sparsity and tensor methods on the task of
training low-memory residual networks; analogs of the schemes also improve the
performance of tensor decomposition techniques. For knowledge distillation,
Frobenius decay enables a simple, overcomplete baseline that yields a compact
model from over-parameterized training without requiring retraining with or
pruning a teacher network. Finally, we show how both schemes applied to
multi-head attention lead to improved performance on both translation and
unsupervised pre-training.
