  Enforcing orthonormal or isometric property for the weight matrices has been
shown to enhance the training of deep neural networks by mitigating gradient
exploding/vanishing and increasing the robustness of the learned networks.
However, despite its practical performance, the theoretical analysis of
orthonormality in neural networks is still lacking; for example, how
orthonormality affects the convergence of the training process. In this letter,
we aim to bridge this gap by providing convergence analysis for training
orthonormal deep linear neural networks. Specifically, we show that Riemannian
gradient descent with an appropriate initialization converges at a linear rate
for training orthonormal deep linear neural networks with a class of loss
functions. Unlike existing works that enforce orthonormal weight matrices for
all the layers, our approach excludes this requirement for one layer, which is
crucial to establish the convergence guarantee. Our results shed light on how
increasing the number of hidden layers can impact the convergence speed.
Experimental results validate our theoretical analysis.
