  A common way to learn and analyze statistical models is to consider
operations in the model parameter space. But what happens if we optimize in the
parameter space and there is no one-to-one mapping between the parameter space
and the underlying statistical model space? Such cases frequently occur for
hierarchical models which include statistical mixtures or stochastic neural
networks, and these models are said to be singular. Singular models reveal
several important and well-studied problems in machine learning like the
decrease in convergence speed of learning trajectories due to attractor
behaviors. In this work, we propose a relative reparameterization technique of
the parameter space, which yields a general method for extracting regular
submodels from singular models. Our method enforces model identifiability
during training and we study the learning dynamics for gradient descent and
expectation maximization for Gaussian Mixture Models (GMMs) under relative
parameterization, showing faster experimental convergence and a improved
manifold shape of the dynamics around the singularity. Extending the analysis
beyond GMMs, we furthermore analyze the Fisher information matrix under
relative reparameterization and its influence on the generalization error, and
show how the method can be applied to more complex models like deep neural
networks.
