  This paper focuses on estimating probability distributions over the set of 3D
rotations ($SO(3)$) using deep neural networks. Learning to regress models to
the set of rotations is inherently difficult due to differences in topology
between $\mathbb{R}^N$ and $SO(3)$. We overcome this issue by using a neural
network to output the parameters for a matrix Fisher distribution since these
parameters are homeomorphic to $\mathbb{R}^9$. By using a negative log
likelihood loss for this distribution we get a loss which is convex with
respect to the network outputs. By optimizing this loss we improve
state-of-the-art on several challenging applicable datasets, namely Pascal3D+,
ModelNet10-$SO(3)$ and UPNA head pose.
