  Estimation of the number of components (or order) of a finite mixture model
is a long standing and challenging problem in statistics. We propose the
Group-Sort-Fuse (GSF) procedure -- a new penalized likelihood approach for
simultaneous estimation of the order and mixing measure in multidimensional
finite mixture models. Unlike methods which fit and compare mixtures with
varying orders using criteria involving model complexity, our approach directly
penalizes a continuous function of the model parameters. More specifically,
given a conservative upper bound on the order, the GSF groups and sorts mixture
component parameters to fuse those which are redundant. For a wide range of
finite mixture models, we show that the GSF is consistent in estimating the
true mixture order and achieves the $n^{-1/2}$ convergence rate for parameter
estimation up to polylogarithmic factors. The GSF is implemented for several
univariate and multivariate mixture models in the R package GroupSortFuse. Its
finite sample performance is supported by a thorough simulation study, and its
application is illustrated on two real data examples.
