  Originally introduced as a neural network for ensemble learning, mixture of
experts (MoE) has recently become a fundamental building block of highly
successful modern deep neural networks for heterogeneous data analysis in
several applications of machine learning and statistics. Despite its popularity
in practice, a satisfactory level of theoretical understanding of the MoE model
is far from complete. To shed new light on this problem, we provide a
convergence analysis for maximum likelihood estimation (MLE) in the
Gaussian-gated MoE model. The main challenge of that analysis comes from the
inclusion of covariates in the Gaussian gating functions and expert networks,
which leads to their intrinsic interaction via some partial differential
equations with respect to their parameters. We tackle these issues by designing
novel Voronoi loss functions among parameters to accurately capture the
heterogeneity of parameter estimation rates. Our findings reveal that the MLE
has distinct behaviors under two complement settings of location parameters of
the Gaussian gating functions, namely when all these parameters are non-zero
versus when at least one among them vanishes. Notably, these behaviors can be
characterized by the solvability of two different systems of polynomial
equations. Finally, we conduct a simulation study to empirically verify our
theoretical results.
