  We introduce a novel approach to perform first-order optimization with
orthogonal and unitary constraints. This approach is based on a parametrization
stemming from Lie group theory through the exponential map. The parametrization
transforms the constrained optimization problem into an unconstrained one over
a Euclidean space, for which common first-order optimization methods can be
used. The theoretical results presented are general enough to cover the special
orthogonal group, the unitary group and, in general, any connected compact Lie
group. We discuss how this and other parametrizations can be computed
efficiently through an implementation trick, making numerically complex
parametrizations usable at a negligible runtime cost in neural networks. In
particular, we apply our results to RNNs with orthogonal recurrent weights,
yielding a new architecture called expRNN. We demonstrate how our method
constitutes a more robust approach to optimization with orthogonal constraints,
showing faster, accurate, and more stable convergence in several tasks designed
to test RNNs.
