  Accurate rotation estimation is at the heart of robot perception tasks such
as visual odometry and object pose estimation. Deep neural networks have
provided a new way to perform these tasks, and the choice of rotation
representation is an important part of network design. In this work, we present
a novel symmetric matrix representation of the 3D rotation group, SO(3), with
two important properties that make it particularly suitable for learned models:
(1) it satisfies a smoothness property that improves convergence and
generalization when regressing large rotation targets, and (2) it encodes a
symmetric Bingham belief over the space of unit quaternions, permitting the
training of uncertainty-aware models. We empirically validate the benefits of
our formulation by training deep neural rotation regressors on two data
modalities. First, we use synthetic point-cloud data to show that our
representation leads to superior predictive accuracy over existing
representations for arbitrary rotation targets. Second, we use image data
collected onboard ground and aerial vehicles to demonstrate that our
representation is amenable to an effective out-of-distribution (OOD) rejection
technique that significantly improves the robustness of rotation estimates to
unseen environmental effects and corrupted input images, without requiring the
use of an explicit likelihood loss, stochastic sampling, or an auxiliary
classifier. This capability is key for safety-critical applications where
detecting novel inputs can prevent catastrophic failure of learned models.
