  Recent research has shown that training low-rank neural networks can
effectively reduce the total number of trainable parameters without sacrificing
predictive accuracy, resulting in end-to-end speedups. However, low-rank model
training necessitates adjusting several additional factorization
hyperparameters, such as the rank of the factorization at each layer. In this
paper, we tackle this challenge by introducing Cuttlefish, an automated
low-rank training approach that eliminates the need for tuning factorization
hyperparameters. Cuttlefish leverages the observation that after a few epochs
of full-rank training, the stable rank (i.e., an approximation of the true
rank) of each layer stabilizes at a constant value. Cuttlefish switches from
full-rank to low-rank training once the stable ranks of all layers have
converged, setting the dimension of each factorization to its corresponding
stable rank. Our results show that Cuttlefish generates models up to 5.6 times
smaller than full-rank models, and attains up to a 1.2 times faster end-to-end
training process while preserving comparable accuracy. Moreover, Cuttlefish
outperforms state-of-the-art low-rank model training methods and other
prominent baselines. The source code for our implementation can be found at:
https://github.com/hwang595/Cuttlefish.
