  Deep linear networks trained with gradient descent yield low rank solutions,
as is typically studied in matrix factorization. In this paper, we take a step
further and analyze implicit rank regularization in autoencoders. We show
greedy learning of low-rank latent codes induced by a linear sub-network at the
autoencoder bottleneck. We further propose orthogonal initialization and
principled learning rate adjustment to mitigate sensitivity of training
dynamics to spectral prior and linear depth. With linear autoencoders on
synthetic data, our method converges stably to ground-truth latent code rank.
With nonlinear autoencoders, our method converges to latent ranks optimal for
downstream classification and image sampling.
