  This paper introduces an extension of the backpropagation algorithm that
enables us to have layers with constrained weights in a deep network. In
particular, we make use of the Riemannian geometry and optimization techniques
on matrix manifolds to step outside of normal practice in training deep
networks, equipping the network with structures such as orthogonality or
positive definiteness. Based on our development, we make another contribution
by introducing the Stiefel layer, a layer with orthogonal weights. Among
various applications, Stiefel layers can be used to design orthogonal filter
banks, perform dimensionality reduction and feature extraction. We demonstrate
the benefits of having orthogonality in deep networks through a broad set of
experiments, ranging from unsupervised feature learning to fine-grained image
classification.
