  We introduce an efficient approach for optimization over orthogonal groups on
highly parallel computation units such as GPUs or TPUs. As in earlier work, we
parametrize an orthogonal matrix as a product of Householder reflections.
However, to overcome low parallelization capabilities of computing Householder
reflections sequentially, we propose employing an accumulation scheme called
the compact WY (or CWY) transform -- a compact parallelization-friendly matrix
representation for the series of Householder reflections. We further develop a
novel Truncated CWY (or T-CWY) approach for Stiefel manifold parametrization
which has a competitive complexity and, again, yields benefits when computed on
GPUs and TPUs. We prove that our CWY and T-CWY methods lead to convergence to a
stationary point of the training objective when coupled with stochastic
gradient descent. We apply our methods to train recurrent neural network
architectures in the tasks of neural machine translation and video prediction.
