  We consider the problem of sampling from posterior distributions for Bayesian
models where some parameters are restricted to be orthogonal matrices. Such
matrices are sometimes used in neural networks models for reasons of
regularization and stabilization of training procedures, and also can
parameterize matrices of bounded rank, positive-definite matrices and others.
In \citet{byrne2013geodesic} authors have already considered sampling from
distributions over manifolds using exact geodesic flows in a scheme similar to
Hamiltonian Monte Carlo (HMC). We propose new sampling scheme for a set of
orthogonal matrices that is based on the same approach, uses ideas of
Riemannian optimization and does not require exact computation of geodesic
flows. The method is theoretically justified by proof of symplecticity for the
proposed iteration. In experiments we show that the new scheme is comparable or
faster in time per iteration and more sample-efficient comparing to
conventional HMC with explicit orthogonal parameterization and Geodesic
Monte-Carlo. We also provide promising results of Bayesian ensembling for
orthogonal neural networks and low-rank matrix factorization.
