  We take a new look at parameter estimation for Gaussian Mixture Models
(GMMs). In particular, we propose using \emph{Riemannian manifold optimization}
as a powerful counterpart to Expectation Maximization (EM). An out-of-the-box
invocation of manifold optimization, however, fails spectacularly: it converges
to the same solution but vastly slower. Driven by intuition from manifold
convexity, we then propose a reparamerization that has remarkable empirical
consequences. It makes manifold optimization not only match EM---a highly
encouraging result in itself given the poor record nonlinear programming
methods have had against EM so far---but also outperform EM in many practical
settings, while displaying much less variability in running times. We further
highlight the strengths of manifold optimization by developing a somewhat tuned
manifold LBFGS method that proves even more competitive and reliable than
existing manifold optimization tools. We hope that our results encourage a
wider consideration of manifold optimization for parameter estimation problems.
